Corpus generation method, corpus generation device, and storage medium
Through sequence labeling, screening and cross-combination of pre-stored corpus data, combined with knowledge graph entities, the problem of long and incomplete corpus data generation cycle in the information retrieval and question-answer system is solved, and more efficient and accurate corpus data generation is achieved.
Patent Information
- Application Number
- CN201911096541.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-11-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2039-12-15
AI Technical Summary
The corpus data generation cycle in the existing information retrieval and question-and-answer system is long and not comprehensive enough, resulting in inefficient answers.
By obtaining pre-stored corpus data for sequence annotation, filtering and grouping, and cross-combining it with knowledge graph entities, generating extended corpus data, shortening the time of corpus data generation and improving data comprehensiveness.
It significantly shortens the time for generating corpus data, makes the generated corpus data more comprehensive and accurate, and improves the response speed and quality of information retrieval and question-answer systems.
Smart Images

Figure CN110852109B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical equipment, and in particular to a corpus generation method, a corpus generation device, and a storage medium. Background Art
[0002] With the development of the Internet, there is an increasing need to obtain answers through online searches. In order to better answer search terms, search terms are usually processed through information retrieval and question-answering systems.
[0003] In the process of information retrieval and question-answering system services, corpus data generation is very important. In related information retrieval and question-answering systems, corpus data generation generally adopts two ways. One is to add and complete through manual search. After manual completion, it needs to be reviewed by developers before it can be put online. The other is through data mining, usually mining logs, using keywords in the field to recall user search statements, and then using models to classify them. After classification, manual annotation is required to see whether it belongs to this field. If it belongs to the user's search statement, the answer is output.
[0004] The above two methods take a long time to generate corpus data, and the generated corpus data is not comprehensive enough. Summary of the invention
[0005] The embodiments of the present invention provide a corpus generation method, a corpus generation device, and a storage medium, which can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0006] An embodiment of the present invention provides a corpus generation method, comprising:
[0007] Acquire pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities;
[0008] Screening the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample;
[0009] Grouping the corpus templates to obtain a plurality of sub-corpus templates;
[0010] Cross-combining multiple sub-corpus templates with entity lists to obtain extended corpus data;
[0011] Combining the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain combined corpus data;
[0012] Target corpus data is generated based on the extended corpus data and the combined corpus data, and the target corpus data is output.
[0013] Accordingly, an embodiment of the present invention further provides a corpus generating device, comprising:
[0014] A tagging unit, used to obtain pre-stored corpus data and perform sequence tagging on the pre-stored corpus data to obtain a tag entity;
[0015] A screening unit, configured to screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample;
[0016] A grouping unit, used for grouping the corpus template to obtain a plurality of sub-corpus templates;
[0017] A cross-combination unit, used for cross-combining multiple sub-corpus templates with entity lists to obtain extended corpus data;
[0018] A combining unit, used to combine the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain combined corpus data;
[0019] A generating unit is used to generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data.
[0020] Optionally, in some embodiments of the present invention, the labeling unit further includes:
[0021] An acquisition unit, configured to acquire annotated corpus data, wherein the annotated corpus data is corpus data that has been annotated;
[0022] A first input unit is used to train the annotated corpus data as training samples to obtain a sequence annotation model;
[0023] The second input unit is used to input the pre-stored corpus data into the sequence annotation model to obtain annotated entities.
[0024] Optionally, in some embodiments of the present invention, the screening unit further comprises:
[0025] A screening subunit, used for screening the subject of the annotated entity so that the screened subject forms a subject text;
[0026] The removal unit is used to remove the annotated entities of the subject text to form a corpus template.
[0027] Optionally, in some embodiments of the present invention, a determining unit is further included.
[0028] The determining unit is specifically used to determine a mapping relationship between the subject text and the output text.
[0029] Optionally, in some embodiments of the present invention, the combination unit further includes:
[0030] A mapping unit, used for mapping the corpus template to an entity list;
[0031] The traversal unit is used to traverse all sub-corpus templates in sequence through the entity list to obtain combined corpus data.
[0032] Optionally, in some embodiments of the present invention, a generating subunit is further included:
[0033] The production subunit is specifically used to generate a question-answer pair text based on the mapping relationship between the subject text and the output text and the combined corpus data.
[0034] Optionally, in some embodiments of the present invention, the combination unit includes:
[0035] A second acquisition unit, used to acquire the knowledge graph entity in the pre-stored corpus data;
[0036] A second removal unit is used to remove entities in the knowledge graph that match the mapping relationship between the subject text and the output text, and obtain the knowledge graph entities after removal;
[0037] The combining subunit is used to combine the removed knowledge graph entity with the corpus template to obtain combined corpus data.
[0038] Optionally, in some embodiments of the present invention, the generating unit includes:
[0039] A completion unit, used for completing the combined corpus data through a completion interface to obtain completed corpus data;
[0040] The second generating subunit is used to generate target corpus data based on the question and answer text and the completed corpus data, and output the target corpus data.
[0041] Correspondingly, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores instructions, and when the instructions are executed by a processor, the steps in any method provided in the embodiment of the present invention are implemented.
[0042] The embodiment of the present invention can obtain pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities, screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample, group the corpus template to obtain multiple sub-corpus templates, cross-combine the multiple sub-corpus templates with an entity list to obtain extended corpus data, combine the knowledge graph entities in the pre-stored corpus data with the corpus template to obtain combined corpus data, generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data. The embodiment of the present invention can shorten the corpus data generation time and make the generated corpus data more comprehensive. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 It is a schematic diagram of an application scenario of the intelligent question-answering system provided by an embodiment of the present invention.
[0045] Figure 2 It is a flowchart of a corpus generation method provided by an embodiment of the present invention.
[0046] Figure 3 It is another flowchart of the corpus generation method provided by an embodiment of the present invention.
[0047] Figure 4 It is a schematic diagram of an application scenario of the corpus generation method provided by an embodiment of the present invention.
[0048] Figure 5 is a schematic diagram of the structure of a corpus generating device provided by an embodiment of the present invention;
[0049] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0051] In the embodiments of the present invention, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts or a combination thereof disclosed in this specification, and are not intended to exclude the possibility of the existence or addition of one or more other features, numbers, steps, behaviors, components, parts or a combination thereof.
[0052] The embodiments of the present invention provide a corpus generation method, a corpus generation device and a storage medium.
[0053] See also Figure 1 , Figure 1 : is a schematic diagram of an application scenario of an intelligent question-answering system provided by an embodiment of the present invention. An embodiment of the present application provides an intelligent question-answering system, which includes an electronic device and a server. The corpus generation device provided by an embodiment of the present invention can be integrated in the electronic device, which can be a mobile phone, a tablet computer, a laptop computer and other devices. In addition, the electronic device can also be connected to other devices, such as servers and other devices. Specifically, the server can include an independently running server, or a distributed server, or a server cluster composed of multiple servers. The server can include a network communication unit, a processor, a memory, and the like. Specifically, the server can provide background services for the above-mentioned electronic devices.
[0054] In actual applications, the intelligent question-answering system is used to support online question-answering services in applications. For example, the intelligent question-answering system supports the question-answering service of Tencent Dingdang. Users enter search statements into the intelligent question-answering system, and the intelligent question-answering system can process the search statements to get answers. In general, the intelligent question-answering system can also include multiple modules to process search statements. Specifically, it includes TASK module, KBQA module, IPQA module and encyclopedia module. For example, when the user's search statement reaches the intelligent question-answering system, it is first classified by the TASK module, which is a multi-classifier, because a search statement may belong to several fields at the same time (for example, "Chengdu" may belong to "geography", "music", "encyclopedia" and other fields at the same time). Then it is sorted by score. If the score of the first comprehensive ranking exceeds the confidence score we set (for example, 0.8), it can be considered that this search statement is a reasonable search statement in the TASK field and should be parsed and processed by the task field. If the score is lower than the threshold, it is distributed downward and processed by KBQA (knowledge-based question-answering service) or IRQA (information retrieval-based question-answering service). If the two models still cannot process the results, they will be handed over to Baidu Encyclopedia for backup processing. If Baidu Encyclopedia cannot process the results, they will be handed over to Xiaoliao for processing, and finally the results will be returned to the user. When using the IRQA service, the generation of corpus data is very important. Using the corpus generation device of the present application to generate corpus data can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0055] The corpus generation method provided by the embodiment of the present invention relates to the natural language processing direction in the field of artificial intelligence.
[0056] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0057] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0058] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.
[0059] It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0060] In one embodiment, the description will be made from the perspective of a corpus generation method, which may be specifically integrated into an electronic device.
[0061] See also Figure 2 As shown, a corpus generation method is provided, which can be executed by an electronic device, and the specific process can be as follows:
[0062] 101. Obtain pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities.
[0063] The pre-stored corpus data is data that already exists in a database, which may be user search logs from a certain period of time (such as one day or one week) of an online application, such as user search logs from Tencent Dingzhong over a period of time. The pre-stored corpus data is annotated to obtain annotated entities.
[0064] Among them, in some embodiments, the “performing sequence annotation on the pre-stored corpus data to obtain annotated entities” specifically includes the steps of:
[0065] (1) Obtaining annotated corpus data, wherein the annotated corpus data is corpus data that has been annotated;
[0066] (2) using the annotated corpus data as training samples to perform training to obtain a sequence annotation model;
[0067] (3) Inputting the pre-stored corpus data into a sequence annotation model to obtain annotated entities.
[0068] The annotated corpus data may come from a massive corpus database in all fields, and the annotated corpus data is the data of the corpus that has been annotated manually in the massive corpus database in all fields. For example, the corpus data is manually annotated with information such as the field and attributes.
[0069] Among them, the annotated corpus data is used as a training sample for training to obtain a sequence annotation model. Because the annotated corpus data has been annotated, the annotated corpus data is used to determine the annotation elements. Then a feature template is defined, and the feature template generates several feature functions based on the marked elements, determines the parameters, and uses the feature functions and the determined parameters to perform model training to obtain a sequence annotation model. The sequence can be annotated according to the sequence annotation model. It should be noted that the sequence annotation model can specifically be a CRF++ model. In the embodiment of the present application, the training of the model will not be described in detail.
[0070] The pre-stored corpus data is input into the sequence annotation model to obtain the annotated entity. The trained corpus annotation model is used to find the most likely hidden state sequence on the new observation sequence, thereby completing the annotation of the pre-stored corpus data and obtaining the annotated entity.
[0071] For example, by performing sequence annotation on corpus data, we can mark the words in the user's search sentence and their corresponding parts of speech, slots, etc. (for example, "Who invented the electric light?" "Electricity": BE (begin entity), "Light": EE (endentity)). We can know that there is an entity here, starting with "electricity" and ending with "light", so this entity is "electric light".
[0072] 102. Screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with extended examples.
[0073] Among them, the subject of the annotated entity is screened, that is, the subject in the annotated entity is screened out, and the annotated entity after removing the subject is the corpus template. The corpus template includes a general text with an extended sample. That is, the sentence pattern of this evening can be extended in the corpus template.
[0074] In some embodiments, the step of “screening the annotated entities to generate a corpus template” specifically includes the following steps:
[0075] (1) screening the subject of the annotated entity so that the screened subject forms a subject text;
[0076] (2) Remove the annotated entities of the subject text to form a corpus template.
[0077] Among them, the subject of the annotated entity is screened, that is, the subject in the annotated entity is screened out, and the annotated entity after removing the subject is the corpus template.
[0078] For example, in the sentence “Who invented the electric light?”, we extract the subject “electric light”, then the subject text is: “electric light”, and the generated corpus template is: “Who invented [entity]?”
[0079] Among them, in some embodiments, the “screening the subject of the annotated entity so that the screened subject forms a subject text” specifically includes the steps of:
[0080] (3) Determine the mapping relationship between the subject text and the output text.
[0081] The mapping relationship between the subject text and the output text is the mapping relationship between the subject and the answer. It is understandable that the mapping relationship between the subject text and the output text can be determined by manual detection. By manually detecting the mapping relationship between the subject text and the output text, wrong answers can be eliminated.
[0082] For example, if the user's search statement is "Who invented the electric light?" and the answer is "Love xx", then the subject is "electric light" and the mapped answer is "Love xx". Dict["electric light"] = "Love xx". For another example, if the user's search statement is "Who invented the electric light?" and the answers are "Love xx" and "Love xxx". We can eliminate the wrong answer "Love xxx".
[0083] 103. Group the corpus templates to obtain multiple sub-corpus templates.
[0084] The corpus modules are grouped to obtain a plurality of sub-corpus modules, and the sub-corpus modules of the same type are divided into intra-group corpus modules, and the intra-group corpus modules include a plurality of corpus modules with the same question.
[0085] For example, [Who invented the electric light] and [Who was the first to invent the electric light], we can divide [Who invented the electric light] and [Who was the first to invent the electric light] into the same group of sub-corpus modules, and the same group of sub-corpus modules can be set as an in-group corpus module.
[0086] For example, the corpus templates for "invention" and "discovery" cannot be grouped together. This is because invention and discovery are different concepts. You can say "Who invented the light bulb" but you cannot say "Who discovered the light bulb". The user's search statement generated in this way is incorrect, so template grouping avoids the generation of incorrect candidate corpora.
[0087] 104. Cross-combining multiple sub-corpus templates with entity lists to obtain extended corpus data.
[0088] Among them, the sub-corpus module and the entity list are cross-combined to obtain several combined corpus templates, and the cross-combined corpus templates are extended corpus data. The extended corpus data can enrich the corpus data, so that the feedback obtained based on the user's search statement is more accurate.
[0089] In some embodiments, the “cross-combining multiple sub-corpus templates with entity lists to obtain extended corpus data” specifically includes the steps of:
[0090] (1) Mapping the corpus template to an entity list;
[0091] (2) Traverse all sub-corpus templates in the entity list in turn to obtain extended corpus data.
[0092] Among them, mapping the corpus template to the entity list is actually extracting the same template and corresponding it to different entities, so as to map the corpus template to the entity list.
[0093] For example: "Who invented the electric light?", "Who invented the umbrella?", "Who invented the air conditioner?"...》Dict["Who invented [entity]?"] = ["electric light", "umbrella", "air conditioner"...]
[0094] The entity list is sequentially traversed through all sub-corpus templates. Specifically, the sub-corpus modules in the same group are cross-combined with the entity list to obtain extended corpus data.
[0095] For example, there are two sub-corpus templates in the same group: Dict["Who invented [entity]"]=["electric light", "umbrella", "air conditioner"...], Dict["Who invented [entity] first"]=["telephone", "computer",...], then ["electric light", "umbrella", "air conditioner"...] will also traverse the template "Who invented [entity] first", ["telephone", "computer",...] will also traverse the template "Who invented [entity]" to generate extended corpus data. In this way, a huge amount of extended corpus data will be generated, and these extended corpus data are all useful.
[0096] Among them, in some embodiments, the “after sequentially traversing the entity list through all sub-corpus templates to obtain extended corpus data” specifically includes the steps of:
[0097] (1) Generate a question-answer text based on the mapping relationship between the subject text and the output text and the combined corpus data.
[0098] Among them, the question and answer text is the input user's search statement that will generate a corresponding answer. Since the mapping relationship between the subject text and the output text has been determined, that is, each subject corresponds to an answer, when the entity list traverses all sub-corpus templates in sequence, the subject used is the answer corresponding to the subject mapping. In this way, we get the generated extended corpus and its corresponding answer.
[0099] It can be understood that in this way, there is no need to manually write answers. When the user enters a search statement, the corresponding answer can be automatically generated.
[0100] 105. Combine the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain combined corpus data.
[0101] It should be noted that knowledge in different fields can have corresponding knowledge graphs (Knowledge Graph / Vault; also known as scientific knowledge graph, called knowledge domain visualization or knowledge domain mapping map in the library and information industry, which is a series of various graphs that show the development process and structural relationship of knowledge, using visualization technology to describe knowledge resources and their carriers, and to mine, analyze, construct, draw and display knowledge and their interrelationships).
[0102] Among them, knowledge graph entities refer to network relationships generated based on multiple nodes. For example, Ai xx invented the electric light, which belongs to the field of electricity, Ai xx belongs to the country of x, and Ai xx created General Electric Co., Ltd.
[0103] Among them, in some embodiments, "combining the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain the combined corpus data" specifically includes the steps of:
[0104] (1) Obtaining knowledge graph entities in the pre-stored corpus data;
[0105] (2) removing entities in the knowledge graph that match the mapping relationship between the subject text and the output text, and obtaining knowledge graph entities after removal;
[0106] (3) Combining the removed knowledge graph entity with the corpus template to obtain combined corpus data.
[0107] Among them, the knowledge graph entities in the pre-stored corpus data are extracted, so that a huge amount of data can be formed. The entities that match the mapping relationship between the subject text and the output text. That is, the answer corresponding to the subject is removed. This can avoid repeated mining of the corpus data that has been mined, and reduce the data carrying load. The removed knowledge graph entity is combined with the corpus template to obtain the combined corpus data.
[0108] For example, the pre-stored corpus data includes who invented the electric light, and the knowledge graph entities about the electric light in the corpus include Love xx, 1979, country x, etc. Since the mapping relationship between Love xx and the electric light has been matched, the knowledge graph entity Love xx is removed, and the remaining knowledge graph entities of 1979 and country x are combined with the corpus template, for example, 1 who invented is combined with 1979, and who invented is combined with country x. Finally, the combined corpus data is obtained.
[0109] 106. Generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data.
[0110] Among them, the method for generating the extended corpus data can greatly relieve the algorithm pressure, and the method for generating the combined corpus data can greatly relieve the data pressure. In this way, the target corpus data generated by this application can relieve both the algorithm pressure and the data pressure. Thus, the corpus data generation time can be shortened, and the generated corpus data can be made more comprehensive.
[0111] Among them, in some embodiments, the “generating target corpus data based on the extended corpus data and the combined corpus data, and outputting the target corpus data” specifically includes the steps of:
[0112] (1) completing the combined corpus data through a completion interface to obtain completed corpus data;
[0113] (2) Generate target corpus data based on the question and answer text and the completed corpus data, and output the target corpus data.
[0114] Among them, Baidu's intelligent completion and mind interface are used to complete the combined corpus data. Baidu's open resources and resources in related fields are used. Because Baidu has billions of queries every day, its expansion words and supplementary vocabulary are relatively complete. We use Baidu's intelligent completion mind interface to optimize the corpus data so that the answer can be automatically completed.
[0115] For example, the search phrase of the representative input user is "Why okra", and the recalled user search phrases are: "Why does okra have mucus", "Why is okra so unpalatable", "Why is okra a bit bitter", "Why are okra seeds bitter", "Why does okra fall after blooming". If they appear frequently in Baidu's search, they pass the verification. The user's search phrase is recalled.
[0116] The question and answer text and the completed corpus data are combined to generate target corpus data, and the target corpus data can be used to automatically supplement the user's search statement and automatically write the answer corresponding to the user's search statement based on the target corpus data.
[0117] The embodiment of the present invention can obtain pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities, screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample, group the corpus template to obtain multiple sub-corpus templates, cross-combine the multiple sub-corpus templates with an entity list to obtain extended corpus data, combine the knowledge graph entities in the pre-stored corpus data with the corpus template to obtain combined corpus data, generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data. The embodiment of the present invention can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0118] like Figure 3 As shown, another corpus generation method is provided, which can be executed by an electronic device, and the specific process can be as follows:
[0119] 201. An electronic device obtains pre-stored corpus data, and performs sequence annotation on the pre-stored corpus data to obtain annotated entities.
[0120] It should be noted that the pre-stored corpus data is data that already exists in the database of the electronic device, and the database may be user search logs from a certain period of time (such as one day or one week) of an online application, such as user search logs of Tencent Dingzhong within a period of time. The pre-stored corpus data is annotated to obtain annotated entities.
[0121] 202. The electronic device performs subject screening on the annotated entity so that the screened subjects form subject texts.
[0122] It should be noted that the electronic device performs subject screening on the annotated entity, that is, screens out the subject in the annotated entity, and the screened out subject is the subject text.
[0123] For example, in the sentence “Who invented the electric light?”, if we extract the subject “electric light”, the subject text will be: “electric light”.
[0124] 203. The electronic device removes the annotated entities of the subject text to form a corpus template.
[0125] It should be noted that the annotated entity of the electronic device after removing the subject is the corpus template.
[0126] For example, in the sentence “Who invented the electric light?”, we extract the subject “electric light” and the generated corpus template is: “Who invented [entity]?”
[0127] 204. The electronic device determines a mapping relationship between the subject text and the output text.
[0128] It should be noted that the electronic device determines the mapping relationship between the subject text and the output text, that is, the mapping relationship between the subject and the answer.
[0129] For example, if the user's search statement is "Who invented the electric light?" and the answer is "Love xx", then the subject is "electric light" and the mapped answer is "Love xx". Dict["electric light"] = "Love xx". For another example, if the user's search statement is "Who invented the electric light?" and the answers are "Love xx" and "Love xxx". We can eliminate the wrong answer "Love xxx".
[0130] 205. The electronic device groups the corpus templates to obtain a plurality of sub-corpus templates.
[0131] It should be noted that the electronic device groups the corpus modules to obtain a plurality of sub-corpus modules, and groups the sub-corpus modules of the same type into intra-group corpus modules, wherein the intra-group corpus modules include a plurality of corpus modules with the same question.
[0132] For example, [Who invented the electric light] and [Who was the first to invent the electric light], we can divide [Who invented the electric light] and [Who was the first to invent the electric light] into the same group of sub-corpus modules, and the same group of sub-corpus modules can be set as an in-group corpus module.
[0133] 206. The electronic device maps the corpus template to an entity list.
[0134] It should be noted that mapping the corpus template to the entity list is actually extracting the same template and corresponding it to different entities, thereby mapping the corpus template to the entity list.
[0135] For example: "Who invented the electric light?", "Who invented the umbrella?", "Who invented the air conditioner?"...》Dict["Who invented [entity]?"] = ["electric light", "umbrella", "air conditioner"...]
[0136] 207. The electronic device sequentially traverses all sub-corpus templates of the entity list to obtain extended corpus data.
[0137] It should be noted that the electronic device sequentially traverses all sub-corpus templates of the entity list, specifically, cross-combines the sub-corpus modules in the same group with the entity list to obtain extended corpus data.
[0138] 208. The electronic device generates a question-answer text based on the mapping relationship between the subject text and the output text and the combined corpus data.
[0139] It should be noted that the question-answer text is the input user's search statement that will generate a corresponding answer. Since the mapping relationship between the subject text and the output text has been determined, that is, each subject corresponds to an answer, when the entity list traverses all sub-corpus templates in sequence, the subject used is the answer corresponding to the subject mapping. In this way, we get the generated extended corpus and its corresponding answer.
[0140] 209. The electronic device obtains the knowledge graph entity in the pre-stored corpus data.
[0141] It should be noted that the electronic device extracts the knowledge graph entities from the pre-stored corpus data, thus forming a huge amount of data.
[0142] 210. The electronic device removes entities in the knowledge graph that match the mapping relationship between the subject text and the output text to obtain the knowledge graph entities after removal.
[0143] It should be noted that the answers corresponding to the subject of electronic equipment are removed. This can avoid repeated mining of the already mined corpus data and reduce the data carrying load.
[0144] 211. The electronic device combines the removed knowledge graph entity with the corpus template to obtain combined corpus data.
[0145] It should be noted that the removed knowledge graph entity is combined with the corpus template to obtain the combined corpus data.
[0146] For example, the pre-stored corpus data includes who invented the electric light, and the knowledge graph entities about the electric light in the corpus include Love xx, 1979, country x, etc. Since the mapping relationship between Love xx and the electric light has been matched, the knowledge graph entity Love xx is removed, and the remaining knowledge graph entities of 1979 and country x are combined with the corpus template, for example, 1 who invented is combined with 1979, and who invented is combined with country x. Finally, the combined corpus data is obtained.
[0147] 212. The electronic device completes the combined corpus data through a completion interface to obtain completed corpus data.
[0148] It should be noted that Baidu's intelligent completion and mind interface are used to complete the combined corpus data. Baidu's open resources and resources in related fields are used. Because Baidu has billions of queries every day, its expansion words and supplementary vocabulary are relatively complete. We use Baidu's intelligent completion mind interface to optimize the corpus data so that the answer can be automatically completed.
[0149] 213. The electronic device generates target corpus data based on the question and answer text and the completed corpus data, and outputs the target corpus data.
[0150] See also Figure 4 The electronic device generates target corpus data from the question and answer text and the completed corpus data, and the target corpus data can be. Based on the target corpus data, we can automatically supplement the user's search statement and automatically write the answer corresponding to the user's search statement.
[0151] The embodiment of the present invention can obtain pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities, screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample, group the corpus template to obtain multiple sub-corpus templates, cross-combine the multiple sub-corpus templates with an entity list to obtain extended corpus data, combine the knowledge graph entities in the pre-stored corpus data with the corpus template to obtain combined corpus data, generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data. The embodiment of the present invention can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0152] In order to facilitate better implementation of the corpus generation method provided in the embodiment of the present invention, the embodiment of the present invention also provides a device based on the above corpus generation method (hereinafter referred to as the generation device). The meaning of the nouns is the same as that in the above corpus generation method, and the specific implementation details can refer to the description in the method embodiment.
[0153] See also Figure 5 , Figure 5 A schematic diagram of the structure of a corpus generation device provided in an embodiment of the present invention, wherein the generation device may include a labeling unit 301, a screening unit 302, a grouping unit 303, a cross-combination unit 30, a combination unit 305, and a generation unit 306, which may be specifically as follows:
[0154] The tagging unit 301 is used to obtain pre-stored corpus data and perform sequence tagging on the pre-stored corpus data to obtain a tag entity;
[0155] A screening unit 302, configured to screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample;
[0156] A grouping unit 303, configured to group the corpus template to obtain a plurality of sub-corpus templates;
[0157] A cross-combination unit 304 is used to cross-combine multiple sub-corpus templates with the entity list to obtain extended corpus data;
[0158] A combining unit 305, configured to combine the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain combined corpus data;
[0159] The generating unit 306 is configured to generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data.
[0160] Optionally, in some embodiments of the present invention, the labeling unit 301 further includes:
[0161] A first acquisition unit is used to acquire annotated corpus data, wherein the annotated corpus data is corpus data that has been annotated;
[0162] A first input unit is used to train the annotated corpus data as training samples to obtain a sequence annotation model;
[0163] The second input unit is used to input the pre-stored corpus data into the sequence annotation model to obtain annotated entities.
[0164] Optionally, in some embodiments of the present invention, the screening unit 302 further includes:
[0165] A screening subunit, used for screening the subject of the annotated entity so that the screened subject forms a subject text;
[0166] The first removal unit is used to remove the annotated entities of the subject text to form a corpus template.
[0167] Optionally, in some embodiments of the present invention, a determining unit 307 is further included.
[0168] The determining unit 307 is specifically configured to determine a mapping relationship between the subject text and the output text.
[0169] Optionally, in some embodiments of the present invention, the combining unit 305 further includes:
[0170] A mapping unit, used for mapping the corpus template to an entity list;
[0171] The traversal unit is used to traverse all sub-corpus templates in sequence through the entity list to obtain combined corpus data.
[0172] Optionally, in some embodiments of the present invention, a generating subunit 308 is further included:
[0173] The production subunit 308 is specifically used to generate a question-answer pair text based on the mapping relationship between the subject text and the output text and the combined corpus data.
[0174] Optionally, in some embodiments of the present invention, the combining unit 305 includes:
[0175] A second acquisition unit, used to acquire the knowledge graph entity in the pre-stored corpus data;
[0176] A second removal unit is used to remove entities in the knowledge graph that match the mapping relationship between the subject text and the output text, and obtain the knowledge graph entities after removal;
[0177] The combining subunit is used to combine the removed knowledge graph entity with the corpus template to obtain combined corpus data.
[0178] Optionally, in some embodiments of the present invention, the generating unit 306 includes:
[0179] A completion unit, used for completing the combined corpus data through a completion interface to obtain completed corpus data;
[0180] The second generating subunit is used to generate target corpus data based on the question and answer text and the completed corpus data, and output the target corpus data.
[0181] The corpus generation device of the present application may include a labeling unit 301, a screening unit 302, a grouping unit 303, a cross-combination unit 30, a combination unit 305, and a generation unit 306, wherein the labeling unit 301 is used to obtain pre-stored corpus data, and perform sequence labeling on the pre-stored corpus data to obtain a labeling entity; the screening unit 302 is used to filter the labeling entity to generate a corpus template, wherein the corpus template includes a general text with an extended sample; the grouping unit 303 is used to group the corpus template to obtain a plurality of sub-corpus templates; the cross-combination unit 304 is used to cross-combine a plurality of sub-corpus templates with an entity list to obtain extended corpus data; the combination unit 305 is used to combine the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain the combined corpus data; the generation unit 306 is used to generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data. The embodiment of the present invention can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0182] Accordingly, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, the electronic device may include a radio frequency (RF) circuit 601, a memory 602 including one or more computer-readable storage media, an input unit 603, a display unit 604, a sensor 605, an audio circuit 606, a wireless fidelity (WiFi) module 607, a processor 608 including one or more processing cores, and a power supply 609. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0183] The RF circuit 601 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 608 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 601 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 601 can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.
[0184] The memory 602 can be used to store software programs and modules. The processor 608 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 608 and the input unit 603 with access to the memory 602.
[0185] The input unit 603 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit 603 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display screen or a touch pad, can collect the user's touch operations on or near it (such as the user's operation on or near the touch-sensitive surface using any suitable object or accessory such as a finger, a stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 608, and can receive and execute the command sent by the processor 608. In addition, the touch-sensitive surface can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch-sensitive surface, the input unit 603 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.
[0186] The display unit 604 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 604 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 608 to determine the type of touch event. The processor 608 then provides a corresponding visual output on the display panel according to the type of touch event. Although in Figure 6 In the embodiment, the touch-sensitive surface and the display panel are used as two independent components to realize input and output functions, but in some embodiments, the touch-sensitive surface and the display panel can be integrated to realize input and output functions.
[0187] The electronic device may also include at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the electronic device is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; As for other sensors that can be configured in electronic devices, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0188] The audio circuit 606, the speaker, and the microphone can provide an audio interface between the user and the electronic device. The audio circuit 606 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 606 and converted into audio data, and then the audio data is output to the processor 608 for processing, and then sent to another electronic device through the RF circuit 601, or the audio data is output to the memory 602 for further processing. The audio circuit 606 may also include an earplug jack to provide communication between an external headset and the electronic device.
[0189] WiFi is a short-range wireless transmission technology. Electronic devices can help users send and receive emails, browse web pages, and access streaming media through WiFi module 607. It provides users with wireless broadband Internet access. Figure 4 A WiFi module 607 is shown, but it is understandable that it is not an essential component of the electronic device and can be omitted as needed without changing the essence of the invention.
[0190] The processor 608 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it executes various functions of the electronic device and processes data, thereby performing overall detection of the mobile phone. Optionally, the processor 608 may include one or more processing cores; preferably, the processor 608 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 608.
[0191] The electronic device also includes a power supply 609 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 608 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 609 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0192] Although not shown, the electronic device may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the processor 608 in the electronic device will load the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 608 will run the applications stored in the memory 602 to achieve various functions:
[0193] Pre-stored corpus data is obtained, and the pre-stored corpus data is sequence-annotated to obtain annotated entities, the annotated entities are screened to generate a corpus template, wherein the corpus template includes a general text with an extended sample, the corpus template is grouped to obtain a plurality of sub-corpus templates, the plurality of sub-corpus templates are cross-combined with an entity list to obtain extended corpus data, the knowledge graph entities in the pre-stored corpus data are combined with the corpus template to obtain combined corpus data, target corpus data is generated based on the extended corpus data and the combined corpus data, and the target corpus data is output. The embodiment of the present invention can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0194] The embodiment of the present invention can obtain pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities, screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample, group the corpus template to obtain multiple sub-corpus templates, cross-combine the multiple sub-corpus templates with an entity list to obtain extended corpus data, combine the knowledge graph entities in the pre-stored corpus data with the corpus template to obtain combined corpus data, generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data. The embodiment of the present invention can shorten the corpus data generation time and make the generated corpus data more comprehensive.
[0195] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0196] To this end, an embodiment of the present invention provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the corpus generation methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:
[0197] An embodiment of the present invention can obtain pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities, screen the annotated entities to generate a corpus template, wherein the corpus template includes a general text with an extended sample, group the corpus template to obtain a plurality of sub-corpus templates, cross-combine the plurality of sub-corpus templates with an entity list to obtain extended corpus data, combine the knowledge graph entities in the pre-stored corpus data with the corpus template to obtain combined corpus data, generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data.
[0198] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0199] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0200] Since the instructions stored in the storage medium can execute the steps in any one of the corpus generation methods provided in the embodiments of the present invention, the beneficial effects that can be achieved by any one of the corpus generation methods provided in the embodiments of the present invention can be achieved. For details, please refer to the previous embodiments and will not be repeated here.
[0201] The corpus generation method, corpus generation device, and storage medium provided in the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A corpus generation method, It is characterized in that include: Acquire pre-stored corpus data, and perform sequence annotation on the pre-stored corpus data to obtain annotated entities; The annotated entities are screened to generate a corpus template, including: screening the annotated entities for subjects so that the screened subjects form subject texts; determining a mapping relationship between the subject texts and output texts; removing the annotated entities of the subject texts to form a corpus template; wherein the corpus template includes a general text with an extended sample; Grouping the corpus templates to obtain a plurality of sub-corpus templates; Cross-combining multiple sub-corpus templates with entity lists to obtain extended corpus data, including: mapping the corpus template to the entity list; traversing the entity list in turn through all sub-corpus templates to obtain extended corpus data; wherein the extended corpus data is the cross-combined corpus template; Combining the knowledge graph entities in the pre-stored corpus data with the corpus template to obtain the combined corpus data, including: obtaining the knowledge graph entities in the pre-stored corpus data; removing the entities in the knowledge graph that match the mapping relationship between the subject text and the output text to obtain the removed knowledge graph entities; combining the removed knowledge graph entities with the corpus template to obtain the combined corpus data; Target corpus data is generated based on the extended corpus data and the combined corpus data, and the target corpus data is output.
2. The corpus generation method according to claim 1, It is characterized in that The step of performing sequence tagging on the pre-stored corpus data to obtain a tag entity includes: Acquire annotated corpus data, wherein the annotated corpus data is corpus data that has been annotated; Using the annotated corpus data as training samples to perform training to obtain a sequence annotation model; The pre-stored corpus data is input into a sequence annotation model to obtain annotated entities.
3. The corpus generation method according to claim 1, It is characterized in that After the entity list is sequentially traversed through all sub-corpus templates to obtain extended corpus data, the method includes: A question-answer pair text is generated based on the mapping relationship between the subject text and the output text and the combined corpus data.
4. The corpus generation method according to claim 3, It is characterized in that The generating target corpus data based on the extended corpus data and the combined corpus data, and outputting the target corpus data, comprises: Completing the combined corpus data through a completion interface to obtain completed corpus data; Generate target corpus data based on the question-answer pair text and the completed corpus data, and output the target corpus data.
5. A corpus generation device, It is characterized in that include: A tagging unit, used to obtain pre-stored corpus data and perform sequence tagging on the pre-stored corpus data to obtain a tag entity; The screening unit is used to screen the annotated entities to generate a corpus template, including: screening the annotated entities for subjects so that the screened subjects form subject texts; determining a mapping relationship between the subject texts and output texts; removing the annotated entities of the subject texts to form a corpus template; wherein the corpus template includes a general text with an extended sample; A grouping unit, used for grouping the corpus template to obtain a plurality of sub-corpus templates; A cross-combination unit is used to cross-combine multiple sub-corpus templates with entity lists to obtain extended corpus data, including: mapping the corpus template to the entity list; traversing the entity list in turn through all sub-corpus templates to obtain extended corpus data; wherein the extended corpus data is the corpus template after cross-combination; A combining unit, used to combine the knowledge graph entity in the pre-stored corpus data with the corpus template to obtain the combined corpus data, including: obtaining the knowledge graph entity in the pre-stored corpus data; removing the entity matching the mapping relationship between the subject text and the output text in the knowledge graph to obtain the knowledge graph entity after removal; combining the knowledge graph entity after removal with the corpus template to obtain the combined corpus data; A generating unit is used to generate target corpus data based on the extended corpus data and the combined corpus data, and output the target corpus data.
6. A storage medium, It is characterized in that The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the corpus generation method according to any one of claims 1 to 4.
7. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the instructions are suitable for being loaded by the processor to execute the steps in the method according to any one of claims 1 to 4.
