Professional domain question and answer pair automatic extraction method and device
By using domain questions and answer extraction models in intelligent question and answer technology, Q&A pairs are automatically extracted from professional field texts, and the high cost and low efficiency problems caused by relying on manual editing in the existing technology are solved, and the professional field intelligence is realized Efficient data processing of Q&A.
Patent Information
- Application Number
- CN202310656004.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
Among the existing intelligent question and answer technologies, the construction of FAQ library in professional fields relies on manual editing, resulting in high cost and low efficiency, and the inability to effectively support intelligent question and answer scenarios in professional fields.
By using domain question extraction models and domain answer extraction models, we automatically extract question-and-answer pairs from professional domain texts, reducing dependence on manual editing and improving data processing efficiency.
It realizes automatic construction of FAQ libraries in professional fields, reduces labor costs, improves data processing efficiency, and solves the high cost and inefficiency problems caused by relying on manual editing in the existing technology.
Smart Images

Figure CN119089905A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent question-answering technology, and in particular to a method and device for automatically extracting professional field question-answer pairs. Background Art
[0002] With the development and popularization of artificial intelligence, intelligent question-answering technology is being used more and more in various industries. In specific intelligent question-answering application scenarios, it usually involves the use of frequently asked questions (FAQ). After receiving the query questions initiated by the user, the FAQ database is searched for the questions closest to the query questions, and the corresponding answers to the questions are returned to the user.
[0003] At present, the general FAQ library is mainly a question-answer library built based on people's daily language conversations, and cannot be directly applied to intelligent question-answering scenarios that support professional fields. And because the professional knowledge involved in different professional fields is different, different FAQ libraries need to be built for each professional field, and they are still mainly built by manual editing. As the demand for question-answering in professional fields increases, the labor cost of building FAQ libraries is very high and the efficiency is low. Summary of the invention
[0004] In view of this, the present application provides a method and device for automatically extracting professional field question and answer pairs, which processes professional field texts by utilizing a field question extraction model and a field answer extraction model, so as to automatically extract field question and answer pairs in the process of using the model to read and process text paragraphs, thereby not relying on labor costs, saving processing costs, and improving data processing efficiency.
[0005] In order to achieve the above objectives, this application mainly provides the following technical solutions:
[0006] The first aspect of the present application provides a method for automatically extracting professional field question-answer pairs, the method comprising:
[0007] Identify the target texts to be processed from the target professional field;
[0008] Processing the target text using a pre-built domain question extraction model to output the domain question corresponding to the target text;
[0009] Using a pre-built domain answer extraction model to process the target text and the domain question, and output a domain answer corresponding to the domain question;
[0010] Based on the domain question and the domain answer, a domain question-answer pair in the target professional domain is composed.
[0011] In some modified implementation manners of the first aspect of the present application, before processing the target text using a preset domain question extraction model and outputting the domain question corresponding to the target text, the method further includes:
[0012] Acquire a first preset semantic model, where the first preset semantic model is a semantic model obtained by training based on general corpus data;
[0013] Based on the first preset semantic model, construct a generative semantic model for extracting summary information from the text corresponding to the target professional field;
[0014] Constructing a first training corpus, wherein the first training corpus is corpus data consisting of sample texts and sample questions corresponding to the sample texts in the target professional field;
[0015] The generative semantic model is trained and adjusted on the first training corpus to obtain a domain problem extraction model used for extracting domain problems from text corresponding to the target professional field.
[0016] In some modified implementations of the first aspect of the present application, the constructing, based on the first preset semantic model, a generative semantic model for extracting summary information from the text corresponding to the target professional field includes:
[0017] Constructing a second training corpus, wherein the second training corpus is corpus data consisting of sample texts and summary information of the sample texts in the target professional field;
[0018] The first preset semantic model is trained and adjusted on the second training corpus to obtain a generative semantic model.
[0019] In some modified implementations of the first aspect of the present application, constructing the second training corpus includes:
[0020] Obtain sample text from the target professional field;
[0021] Among the multiple sentences included in the sample text, a preset text summary evaluation algorithm is used to calculate an evaluation score of the semantic similarity between each sentence and the text composed of the remaining sentences;
[0022] Based on the ranking of the evaluation scores of the sentences, a preset number of target sentences are obtained from the sample text to form summary information corresponding to the sample text;
[0023] The remaining sentences in the sample text except the target sentence form a target text paragraph;
[0024] The second training corpus is constructed according to the summary information corresponding to each of the target text paragraphs.
[0025] In some modified implementations of the first aspect of the present application, before the target text and the domain question are processed by the pre-built domain answer extraction model to output the domain answer corresponding to the domain question, the method further includes:
[0026] Acquire a second preset semantic model, where the second preset semantic model is a semantic model obtained by training based on general corpus data;
[0027] Constructing a third training corpus, wherein the third training corpus at least includes: sample texts in the target professional field, sample questions and sample answers corresponding to the sample texts, and the sample texts and the sample questions serve as input data for model training, and the sample answers serve as output data for model training;
[0028] The second preset semantic model is trained and adjusted on the third training corpus to obtain a domain answer extraction model for extracting domain answers from text corresponding to the target professional field.
[0029] In some modified implementations of the first aspect of the present application, the domain question and answer pair in the target professional field is formed based on the domain question and the corresponding domain answer, including:
[0030] Establishing a mapping relationship between the domain question and the domain answer according to the semantic association relationship between the domain question and the domain answer;
[0031] According to the mapping relationship, a domain question and answer pair is formed based on the domain question and the domain answer corresponding to the domain question.
[0032] In some modified implementation modes of the first aspect of the present application, after forming the domain question-answer pair, the method further includes:
[0033] For each domain answer in the domain question-answer pair, based on the location of the domain answer in the target text, obtaining a target sentence containing the domain answer;
[0034] The content information contained in the target sentence is used to expand the content information of the domain question-answer pair.
[0035] The second aspect of the present application provides a device for automatically extracting professional field question-answer pairs, the device comprising:
[0036] A determination unit, used for determining a target text to be processed from a target professional field;
[0037] A first processing unit is used to process the target text using a pre-built domain problem extraction model, and output the domain problem corresponding to the target text;
[0038] A second processing unit is used to process the target text and the domain question using a pre-built domain answer extraction model, and output a domain answer corresponding to the domain question;
[0039] A composition unit is used to compose a domain question-answer pair in the target professional domain based on the domain question and the domain answer.
[0040] In some modified implementation manners of the second aspect of the present application, before the target text is processed by using a preset domain problem extraction model to output the domain problem corresponding to the target text, the device further includes:
[0041] An acquisition unit, configured to acquire a first preset semantic model, wherein the first preset semantic model is a semantic model obtained by training based on general corpus data;
[0042] A first construction unit is used to construct a generative semantic model for extracting summary information from the text corresponding to the target professional field based on the first preset semantic model;
[0043] A second construction unit is used to construct a first training corpus, wherein the first training corpus is corpus data consisting of sample texts and sample questions corresponding to the sample texts in the target professional field;
[0044] The first training unit is used to train and adjust the generative semantic model on the first training corpus to obtain a domain problem extraction model used for extracting domain problems from texts corresponding to the target professional field.
[0045] In some modified implementation modes of the second aspect of the present application, the first building unit includes:
[0046] A construction module, used for constructing a second training corpus, wherein the second training corpus is corpus data consisting of sample texts and summary information of the sample texts in the target professional field;
[0047] A training module is used to train and adjust the first preset semantic model on the second training corpus to obtain a generative semantic model.
[0048] In some modified implementation modes of the second aspect of the present application, the building blocks are further specifically used for:
[0049] Obtain sample text from the target professional field;
[0050] Among the multiple sentences included in the sample text, a preset text summary evaluation algorithm is used to calculate an evaluation score of the semantic similarity between each sentence and the text composed of the remaining sentences;
[0051] Based on the ranking of the evaluation scores of the sentences, a preset number of target sentences are obtained from the sample text to form summary information corresponding to the sample text;
[0052] The remaining sentences in the sample text except the target sentence form a target text paragraph;
[0053] The second training corpus is constructed according to the summary information corresponding to each of the target text paragraphs.
[0054] In some modified implementations of the second aspect of the present application, before the target text and the domain question are processed by using the pre-built domain answer extraction model to output the domain answer corresponding to the domain question, the device further includes:
[0055] The acquisition unit is further used to acquire a second preset semantic model, where the second preset semantic model is a semantic model obtained by training based on general corpus data;
[0056] The third construction unit is further used to construct a third training corpus, wherein the third training corpus at least includes: sample texts in the target professional field, sample questions and sample answers corresponding to the sample texts, and the sample texts and the sample questions serve as input data for model training, and the sample answers serve as output data for model training;
[0057] The second training unit is used to train and adjust the second preset semantic model on the third training corpus to obtain a domain answer extraction model used to extract domain answers from the text corresponding to the target professional field.
[0058] In some modified implementation modes of the second aspect of the present application, the constituent unit includes:
[0059] An establishment module is used to establish a mapping relationship between the domain question and the domain answer according to the semantic association relationship between the domain question and the domain answer;
[0060] A composition module is used to compose a domain question-answer pair based on the mapping relationship, the domain question and the domain answer corresponding to the domain question.
[0061] In some modified implementation modes of the second aspect of the present application, after the domain question-answer pair is formed, the constituent unit further includes:
[0062] An acquisition module, configured to acquire, for each domain answer in a domain question-answer pair, a target sentence containing the domain answer based on the location of the domain answer in the target text;
[0063] The expansion module is used to expand the content information of the domain question and answer pair by utilizing the content information contained in the target sentence.
[0064] A third aspect of the present application provides a storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above-mentioned method for automatically extracting professional field question and answer pairs.
[0065] A fourth aspect of the present application provides an electronic device, the device comprising at least one processor, and at least one memory and a bus connected to the processor;
[0066] Wherein, the processor and the memory communicate with each other through the bus;
[0067] The processor is used to call the program instructions in the memory to execute the above-mentioned method for automatically extracting professional field question and answer pairs.
[0068] By means of the above technical solution, the technical solution provided by this application has at least the following advantages:
[0069] The present application provides a method and device for automatically extracting professional field question-answer pairs. For the target text to be processed in the target professional field, the present application first uses the field question extraction model to extract field questions from the target text, and then uses the field answer extraction model to process the target text and the previously extracted field questions, outputs the field answers corresponding to the field questions, and finally forms field question-answer pairs in the target professional field based on the semantic relevance of the obtained field questions and field answers. Compared with the prior art, the technical problem of relying on manual editing to build FAQ libraries, which leads to high costs and low efficiency, is solved. The present application mainly uses two models to automatically complete the extraction of questions and answers in professional fields, thereby not relying on labor costs, saving processing costs, and improving data processing efficiency.
[0070] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0072] Figure 1 A flow chart of a method for automatically extracting professional field question and answer pairs provided in an embodiment of the present application;
[0073] Figure 2 A flow chart of another method for automatically extracting professional field question and answer pairs provided in an embodiment of the present application;
[0074] Figure 3 A block diagram of a device for automatically extracting professional field question-answer pairs provided in an embodiment of the present application;
[0075] Figure 4 A block diagram of another apparatus for automatically extracting professional field question-answer pairs provided in an embodiment of the present application;
[0076] Figure 5 An electronic device for automatically extracting professional field question and answer pairs provided in an embodiment of the present application. DETAILED DESCRIPTION
[0077] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0078] Now, for the frequently asked questions (FAQ) library used in the intelligent question-and-answer application scenario, if it does not involve professional fields, that is, professional field knowledge is not required, a general FAQ library can be used to meet the user's daily question-and-answer needs.
[0079] However, the inventors found that if it is necessary to involve professional fields, and because there is still a knowledge gap between different professional fields, it is necessary to build a corresponding FAQ library for each professional field. Due to the reliance on the professional nature of knowledge, more labor costs are now required to complete the construction. As the number of professional fields increases, this will inevitably increase labor costs, and efficiency will also decrease accordingly.
[0080] To this end, after research, the inventors have found that in order to ensure the quality of constructing a professional field FAQ library and improve processing efficiency, the preferred solution is to automatically extract field questions and field answers in the process of automatically reading the target text in the target professional field to construct field question and answer pairs, thereby automatically completing the construction of the FAQ library corresponding to the target professional field.
[0081] Based on the above considerations, this application provides a method for automatically extracting professional field question and answer pairs, such as Figure 1 , the present application embodiment provides the following specific steps:
[0082] 101. Identify the target text to be processed from the target professional field.
[0083] 102. Use the pre-built domain question extraction model to process the target text and output the domain questions corresponding to the target text.
[0084] Among them, the domain question extraction model can be trained and adjusted based on an open source modern semantic model to obtain a model that can be applied to professional fields for semantic understanding and question extraction.
[0085] For example, based on a semantic model that can be used for text generation, such as the MT5 model, a multilingual version of the T5 model, the domain question extraction model required by the embodiment of the present application is trained.
[0086] The function achieved by the domain problem extraction model obtained through training in the embodiment of the present application is to input a target text in a target professional field, and output corresponding domain problems for the target text during the reading process through the domain problem extraction model.
[0087] 103. Use the pre-built domain answer extraction model to process the target text and domain questions, and output the domain answers corresponding to the domain questions.
[0088] Among them, the domain answer extraction model can be trained and adjusted based on an open source modern semantic model to obtain a model that can be applied to professional fields for semantic understanding and answer extraction.
[0089] For example, a model based on Chinese semantic understanding, such as the Bert model, is used to train the domain answer extraction model required by the embodiments of the present application.
[0090] The function implemented by the domain answer extraction model trained in the embodiment of the present application is to input a target text in a target professional field and a domain question raised for the target text, and output a domain answer to the domain question during the reading process through the domain answer extraction model.
[0091] 104. Based on domain questions and domain answers, form domain question and answer pairs in the target professional field.
[0092] In the embodiments of the present application, the domain question extraction model and the domain answer extraction model can process multiple target texts in the target professional field in batch operations. For the large number of domain questions and domain answers outputted, there is a certain correlation in terms of semantic relevance, that is, one domain question can correspond to one or more domain answers, thereby forming a domain question and answer pair based on each domain question, which is used to construct a FAQ library in the target professional field.
[0093] As described above, the embodiment of the present application provides a method for automatically extracting professional field question-answer pairs. For the target text to be processed in the target professional field, the embodiment of the present application first uses the domain question extraction model to extract domain questions from the target text, and then uses the domain answer extraction model to process the target text and the previously extracted domain questions, outputs the domain answers corresponding to the domain questions, and finally composes domain question-answer pairs in the target professional field based on the semantic relevance of the obtained domain questions and domain answers. Compared with the prior art, the technical problem of relying on manual editing to build FAQ libraries, which leads to high costs and low efficiency, is solved. The embodiment of the present application mainly uses two models to automatically complete the extraction of professional field questions and answers, thereby not relying on labor costs, saving processing costs, and improving data processing efficiency.
[0094] In order to explain the above embodiment in more detail, the present application embodiment also provides another method for automatically extracting professional field question and answer pairs, such as Figure 2 As shown, the present application embodiment provides the following specific steps:
[0095] 201. Obtain a first preset semantic model, where the first preset semantic model is a semantic model obtained through training based on general corpus data.
[0096] In the embodiment of the present application, the domain problem extraction model is trained based on the first preset semantic model (e.g., MT5 model). The first preset semantic model is a semantic model obtained by training based on general corpus data. Although it does not have the ability to process text paragraphs with professional domain knowledge, it has the ability to understand the semantics of everyday language (i.e., non-professional domain language). Therefore, the embodiment of the present application performs training adjustments based on this model to save more unnecessary processing costs (e.g., there is no need to spend the cost of retraining a semantic model from scratch), so as to more efficiently train a model for extracting domain problems from texts applied to professional fields. The specific detailed steps for training the domain problem extraction model are as follows: steps 202-204.
[0097] 202. Based on the first preset semantic model, construct a generative semantic model for extracting summary information from text corresponding to the target professional field.
[0098] It should be noted that the words "first" and "second" are used in the embodiments of the present application only for the convenience of distinguishing and referring to different preset semantic models, and there is no other ambiguity such as sorting.
[0099] The function implemented by the generative semantic model constructed in the embodiment of the present application is to input a target text in a target professional field, and output summary information of the target text during the reading process of the generative semantic model. In order to enable the generative semantic model to have such a function, the embodiment of the present application provides the following training process, which specifically includes the following:
[0100] First, a training corpus is constructed, which is corpus data consisting of sample texts and summary information of the sample texts in the target professional field. Second, a first preset semantic model is trained and adjusted on such training corpus to obtain a generative semantic model.
[0101] Example 1: Take a piece of corpus information as {"source_text":"AAA","target_text":"BBB"}, where "source_text" is the input information and "target_text" is the output information. As follows:
[0102] {"source_text":"The relationship between faults and oil and gas migration is re-understood by applying the idea of fault-controlled reservoirs. The relationship between growth faults and oil and gas accumulation in Wenliu Oil and Gas Field is studied by zones and types, which is conducive to further overall evaluation of the oil and gas exploration potential of Wenliu Oil and Gas Field and the search for new oil and gas enriched blocks. This paper uses techniques such as fault activity rate and thermal simulation of hydrocarbon generation history to study and explain the development and evolution history of each fault system in Wenliu Oil and Gas Field and the control of the cross-section morphology of the second and third order faults in Wenliu area on the distribution of oil and gas resources. It is believed that the intensity of fault activity during the hydrocarbon generation and expulsion period and the dominant migration channel of the cross section control the distribution of oil and gas.","target_text":"The Wenliu Oil and Gas Field has strong tectonic activity, mainly developing extensional syngenetic normal faults. The development of second and third order faults is the main channel for the vertical migration of mature oil and gas in the basin. The enrichment of oil and gas is closely related to growth faults."}
[0103] As mentioned above, for such a piece of corpus information, it includes: a text paragraph as input information, and the summary information corresponding to the text paragraph as output information. Based on a large amount of corpus, the embodiment of the present application utilizes such model input information and model output information to continuously optimize and adjust the first preset semantic model so that the model finally trained has the function of generating summary information from professional field texts.
[0104] Furthermore, for constructing such a training corpus, a preferred implementation method is to adopt the following automated construction method, which specifically includes the following:
[0105] First, a sample text is obtained from the target professional field. Among the multiple sentences contained in the sample text, a preset text summary evaluation algorithm is used to calculate the evaluation score of the semantic similarity between each sentence and the text composed of the remaining sentences.
[0106] Among them, the preset text summary evaluation algorithm is preferably a Neuro-Linguistic Programming (NLP) Rouge_L score calculation algorithm, so as to evaluate the semantic similarity score between each sentence and the text composed of the remaining sentences, that is, to evaluate whether the meaning of each sentence is sufficient to describe the subject information of the text one by one. If the evaluation score is higher, it indicates that the sentence can be used as part of the summary information.
[0107] Secondly, based on the ranking of the evaluation scores of each sentence, a preset number of target sentences are obtained from the sample text to form summary information corresponding to the sample text. The remaining sentences in the sample text except the target sentences are formed into target text paragraphs, and the second training corpus is constructed based on the summary information corresponding to each target text paragraph.
[0108] Exemplarily, for any sample text in the target professional field, the embodiments of the present application can be processed by a preset text summary evaluation algorithm. In a sample text, the preferred implementation method is to screen out a preset number of sentences with high evaluation scores. Alternatively, the sentences are screened based on the proportion of all sentences. For example, in a sample text, the n / 4 sentences with the highest scores are selected to be organized into summary information as the "target_text" information in a corpus, and the remaining 3n / 4 sentences are formed into ordinary text paragraphs as a whole as the "source_text" information in the corpus. Accordingly, each sample text in the target professional field is automatically divided into "target_text" and "source_text". As the number of sample texts increases, a large amount of training corpus suitable for semantic processing in the target professional field is obtained.
[0109] 203. Construct a training corpus, where the training corpus is corpus data consisting of sample texts and sample questions corresponding to the sample texts in the target professional field.
[0110] After obtaining the generative semantic model basis in step 202, based on the model's function of processing output summary information, the embodiment of the present application also performs training adjustment so that the trained model has the function of processing output domain problems. First, the training corpus required for the training adjustment is constructed. The training corpus is corpus data consisting of sample texts and sample problems corresponding to the sample texts in the target professional field.
[0111] In a preferred implementation, a reading comprehension dataset is collected from public datasets to form a question generation dataset as a training corpus, which is used to train and adjust a generative semantic model to obtain a domain question extraction model.
[0112] Example 2, citing a piece of corpus information as {"source_text":"CCC","target_text":"DDD"}, where "source_text" is input information, "target_text" is output information, and "source_text" is sample text and "target_text" is sample question.
[0113] For example, {"source_text":"Jiangsu Road Subdistrict is a subdistrict office under the jurisdiction of Changning District, Shanghai, China. It is located in the easternmost part of Changning District. It borders on Jing'an Temple Subdistrict in Jing'an District to the east with Zhenning Road, Caojiadu Subdistrict in Jing'an District to the north with Wuding West Road, and Hunan Road Subdistrict in Xuhui District to the south with Huashan Road. The western boundaries are Changning Road, Anxi Road, Wuyi Road, etc. It covers an area of 1.52 square kilometers, has a registered population of 52,600, and has 13 neighborhood committees under its jurisdiction. The Changning District Government is located on Yuyuan Road (near Anxi Road) within the jurisdiction of the subdistrict. The main streets of Jiangsu Road Subdistrict, Jiangsu Road, Yuyuan Road, Changning Road, Wuyi Road, Wuding West Road, and Yan'an West Road, are blocks with concentrated garden houses and are the historical and cultural features of Yuyuan Road. The main part of the street. The more famous modern buildings include the Chinese and Western Girls' High School (now Shanghai No. 3 Girls' High School) and Xiyuan Apartments. Jiangsu Road, Changning Road, Yan'an West Road (elevated) and other roads have been widened to form Shanghai's main traffic arteries. Shanghai Metro Line 2 passes through the street and has a Jiangsu Road Station. Jiangsu Road Street has jurisdiction over Qishan Residential Committee, Jiangsu Residential Committee, Wancun Residential Committee, Nanwang Residential Committee, Dongbang Residential Committee, Yusan Residential Committee, Caojiayan Residential Committee, Xibang Residential Committee, Fushi Residential Committee, Changxin Residential Committee, Huashan Residential Committee, Lixi Residential Committee, and Beiwang Residential Committee. ","target_text":"Where is Jiangsu Road Street in Shanghai? [SEP]How many neighborhood committees are under the jurisdiction of Jiangsu Road Street? "}
[0114] 204. Train and adjust the generative semantic model on the training corpus to obtain a domain problem extraction model for extracting domain problems from texts corresponding to the target professional field.
[0115] The embodiment of the present application is based on the generative semantic model and uses "corpus data consisting of sample texts and sample questions corresponding to the sample texts in the target professional field" for re-training and adjustment, which is equivalent to upgrading the function of processing and outputting summary information to also have the function of converting summary information into questions, that is, for the domain problem extraction model trained and adjusted on the generative semantic model, its process of reading and processing the target text in the target professional field is actually asking questions based on the obtained summary information, thereby outputting the domain problems corresponding to the target text.
[0116] The above is a specific implementation method for constructing a domain question extraction model based on steps 201-204 provided in the embodiment of the present application. Below, as shown in steps 205-207, the embodiment of the present application also provides a specific implementation method for constructing a domain answer extraction model, which is explained as follows.
[0117] 205. Obtain a second preset semantic model, where the second preset semantic model is a semantic model obtained through training based on general corpus data.
[0118] In the embodiment of the present application, the second preset semantic model (e.g., Bert model) is used as the basis to train the domain answer extraction model, which has the function of semantic understanding of Chinese, but does not have the ability to process text paragraphs with professional domain knowledge. The embodiment of the present application performs training adjustments based on this model to save more unnecessary processing costs (e.g., there is no need to spend the cost of retraining a semantic model from scratch), so as to more efficiently train a model for extracting domain answers from texts applied to professional fields. The specific detailed steps for training the domain answer extraction model are as follows: steps 206-207.
[0119] 206. Construct a training corpus, which includes at least: sample texts in the target professional field, sample questions and sample answers corresponding to the sample texts, and the sample texts and sample questions serve as input data for model training, and the sample answers serve as output data for model training.
[0120] The preferred implementation method is to collect reading comprehension datasets from public datasets. The format of a reading comprehension dataset is as follows: {"context":"Jiangsu Road Subdistrict is a subdistrict office under the jurisdiction of Changning District, Shanghai, China. It is located in the easternmost part of Changning District. It is adjacent to Jing'an Temple Subdistrict in Jing'an District to the east of Zhenning Road, Caojiadu Subdistrict in Jing'an District to the north of Wuding West Road, and Hunan Road Subdistrict in Xuhui District to the south of Huashan Road. The western boundaries are Changning Road, Anxi Road, Wuyi Road, etc. It covers an area of 1.52 square kilometers, has a registered population of 52,600, and has 13 neighborhood committees under its jurisdiction. The Changning District Government is located on Yuyuan Road (near Anxi Road) within the jurisdiction of the subdistrict. The main streets of Jiangsu Road Subdistrict are Jiangsu Road, Yuyuan Road, and Yuyuan Road. Yuyuan Road, Changning Road, Wuyi Road, Wuding West Road, and Yan'an West Road are blocks with concentrated garden houses and are the main part of the historical and cultural area of Yuyuan Road. The more famous modern buildings include Zhongxi Girls' High School (now Shanghai No. 3 Girls' High School) and Xiyuan Apartment. Jiangsu Road, Changning Road, and Yan'an West Road (elevated) have been widened to form the main traffic arteries of Shanghai. Shanghai Metro Line 2 passes through the jurisdiction of this street and has a Jiangsu Road Station. Jiangsu Road Street has Qishan Residents Committee, Jiangsu Residents Committee, Wancun Residents Committee, Nanwang Residents Committee, Dongbang Residents Committee, Yusan Residents Committee, Caojiayan Residents Committee, Xibang Residents Committee, Fushi Residents Committee, Changxin Residents Committee, Huashan Residents Committee, Lixi Residents Committee, and Beiwang Residents Committee. ",
[0121]
[0122]
[0123] Based on such a reading comprehension dataset, a reading comprehension corpus can be constructed as a training corpus. For example, a piece of corpus information can be expressed as follows:
[0124] Model input information is: {"context"} <sep>{"question"};
[0125] The model output information is: {"answer_start"} (that is, the starting position of the answer in "context") and the length of the string {"answer_start"} + {"text"} (that is, the ending position of the answer in "context").
[0126] 207. Train and adjust the second preset semantic model on such training corpus to obtain a domain answer extraction model for extracting domain answers from texts corresponding to the target professional field.
[0127] Based on the training corpus obtained in step 206, the second preset semantic model is continuously trained and adjusted so that the model realizes the function of inputting a target text in the target professional field and domain questions raised for the target text, and outputting domain answers during the reading process.
[0128] 208. Determine the target text to be processed from the target professional field.
[0129] 209. Use the pre-built domain problem extraction model to process the target text and output the domain problems corresponding to the target text.
[0130] 210. Use the pre-built domain answer extraction model to process the target text and domain questions, and output the domain answers corresponding to the domain questions.
[0131] In the embodiment of the present application, for the explanation of steps 208-210, please refer to steps 201-203, which will not be repeated here.
[0132] 211. Based on domain questions and domain answers, form domain question and answer pairs in the target professional field.
[0133] In the embodiment of the present application, this step can be further refined to include: establishing a mapping relationship between the domain question and the domain answer based on the semantic association relationship between the domain question and the domain answer, and forming a domain question and answer pair based on the domain question and the domain answer corresponding to the domain question according to the mapping relationship, that is, obtaining one or more domain answers corresponding to a domain question.
[0134] Furthermore, in order to enrich the content information contained in the domain answer, the embodiment of the present application obtains the target sentence containing the domain answer for each domain question-answer pair based on the location of the domain answer in the target text, so as to use the content information contained in the target sentence to expand the content information of the domain answer. Then, if a domain question corresponds to multiple domain answers, richer answer information will be obtained in response to the domain question.
[0135] Furthermore, as a response to the above Figure 1 , Figure 2 The implementation of the method shown in the figure, the embodiment of the present application provides a device for automatically extracting professional field question and answer pairs. The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, the device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. The device is used to extract field question and answer pairs from professional field texts to construct a FAQ library, specifically as follows Figure 3 As shown, the device comprises:
[0136] A determination unit 301 is used to determine a target text to be processed from a target professional field;
[0137] The first processing unit 302 is used to process the target text using a pre-built domain question extraction model and output the domain question corresponding to the target text;
[0138] The second processing unit 303 is used to process the target text and the domain question using a pre-built domain answer extraction model, and output a domain answer corresponding to the domain question;
[0139] The composition unit 304 is used to compose a domain question-answer pair in the target professional domain based on the domain question and the domain answer.
[0140] Further, such as Figure 4 As shown, before the target text is processed by using the preset domain problem extraction model and the domain problem corresponding to the target text is output, the device further includes:
[0141] An acquisition unit 305 is used to acquire a first preset semantic model, where the first preset semantic model is a semantic model obtained by training based on general corpus data;
[0142] A first construction unit 306 is used to construct a generative semantic model for extracting summary information from the text corresponding to the target professional field based on the first preset semantic model;
[0143] A second construction unit 307 is used to construct a first training corpus, wherein the first training corpus is corpus data consisting of sample texts and sample questions corresponding to the sample texts in the target professional field;
[0144] The first training unit 308 is used to train and adjust the generative semantic model on the first training corpus to obtain a domain problem extraction model used for extracting domain problems from texts corresponding to the target professional field.
[0145] Further, such as Figure 4 As shown, the first construction unit 306 includes:
[0146] A construction module 3061 is used to construct a second training corpus, where the second training corpus is corpus data consisting of sample texts and summary information of the sample texts in the target professional field;
[0147] The training module 3062 is used to train and adjust the first preset semantic model on the second training corpus to obtain a generative semantic model.
[0148] Further, such as Figure 4 As shown, the construction module 3061 is also specifically used for:
[0149] Obtain sample text from the target professional field;
[0150] Among the multiple sentences included in the sample text, a preset text summary evaluation algorithm is used to calculate an evaluation score of the semantic similarity between each sentence and the text composed of the remaining sentences;
[0151] Based on the ranking of the evaluation scores of the sentences, a preset number of target sentences are obtained from the sample text to form summary information corresponding to the sample text;
[0152] The remaining sentences in the sample text except the target sentence form a target text paragraph;
[0153] The second training corpus is constructed according to the summary information corresponding to each of the target text paragraphs.
[0154] Further, such as Figure 4 As shown, before the target text and the domain question are processed by the pre-built domain answer extraction model and the domain answer corresponding to the domain question is output, the device further includes:
[0155] The acquisition unit 305 is further used to acquire a second preset semantic model, where the second preset semantic model is a semantic model obtained by training based on general corpus data;
[0156] The third construction unit 309 is further used to construct a third training corpus, wherein the third training corpus at least includes: sample texts in the target professional field, sample questions and sample answers corresponding to the sample texts, and the sample texts and the sample questions are used as input data for model training, and the sample answers are used as output data for model training;
[0157] The second training unit 310 is used to train and adjust the second preset semantic model on the third training corpus to obtain a domain answer extraction model used to extract domain answers from text corresponding to the target professional field.
[0158] Further, such as Figure 4 As shown, the component unit 304 includes:
[0159] Establishing module 3041, used to establish a mapping relationship between the domain question and the domain answer according to the semantic association relationship between the domain question and the domain answer;
[0160] The composition module 3042 is used to compose a domain question-answer pair based on the mapping relationship, the domain question and the domain answer corresponding to the domain question.
[0161] Further, such as Figure 4 As shown, after forming the domain question-answer pair, the forming unit 34 further includes:
[0162] An acquisition module 3043 is used to acquire, for each domain answer in the domain question-answer pair, a target sentence containing the domain answer based on the location of the domain answer in the target text;
[0163] The expansion module 3044 is used to expand the content information of the domain question and answer pair by utilizing the content information contained in the target sentence.
[0164] In summary, the embodiments of the present application provide a method and device for automatically extracting professional field question-answer pairs. For the target text to be processed in the target professional field, the present application first uses the domain question extraction model to extract domain questions from the target text, and then uses the domain answer extraction model to process the target text and the previously extracted domain questions, outputs the domain answers corresponding to the domain questions, and finally composes domain question-answer pairs in the target professional field based on the semantic relevance of the obtained domain questions and domain answers. Compared with the prior art, the technical problem of relying on manual editing to build a FAQ library, which leads to high costs and low efficiency, is solved. The present application mainly uses two models to automatically complete the extraction of professional field questions and answers, thereby not relying on labor costs, saving processing costs, and improving data processing efficiency.
[0165] The automatic extraction device for professional field question and answer pairs includes a processor and a memory. The above-mentioned determination unit, first processing unit, second processing unit and composition unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.
[0166] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the domain question extraction model and the domain answer extraction model can be used to process professional domain texts by adjusting the kernel parameters. In the process of using the model to read and process text paragraphs, domain question and answer pairs are automatically extracted, thereby not relying on labor costs, saving processing costs, and improving data processing efficiency.
[0167] An embodiment of the present application provides a storage medium on which a program is stored. When the program is executed by a processor, the method for automatically extracting professional field question and answer pairs is implemented.
[0168] An embodiment of the present application provides a processor, which is used to run a program, wherein the program executes the method for automatically extracting professional field question and answer pairs when running.
[0169] The present application embodiment provides an electronic device 4, such as Figure 5 As shown, the device includes at least one processor 41, and at least one memory 42 and a bus 43 connected to the processor 41; wherein the processor 41 and the memory 42 communicate with each other via the bus 43; the processor 41 is used to call the program instructions in the memory 42 to execute the above-mentioned method for automatically extracting professional field question and answer pairs.
[0170] The device in this article can be a server, PC, etc.
[0171] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of initializing the method for automatically extracting question-answer pairs in professional fields.
[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0173] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.
[0174] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
[0175] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0176] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.
[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0178] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.< / sep>
Claims
1. A method for automatically extracting professional field question-answer pairs, characterized in that: The method comprises: Identify the target texts to be processed from the target professional field; Processing the target text using a pre-built domain question extraction model to output the domain question corresponding to the target text; Using a pre-built domain answer extraction model to process the target text and the domain question, and output a domain answer corresponding to the domain question; Based on the domain question and the domain answer, a domain question-answer pair in the target professional domain is composed.
2. The method according to claim 1, characterized in that Before processing the target text using the preset domain problem extraction model and outputting the domain problem corresponding to the target text, the method further includes: Acquire a first preset semantic model, where the first preset semantic model is a semantic model obtained by training based on general corpus data; Based on the first preset semantic model, construct a generative semantic model for extracting summary information from the text corresponding to the target professional field; Constructing a first training corpus, wherein the first training corpus is corpus data consisting of sample texts and sample questions corresponding to the sample texts in the target professional field; The generative semantic model is trained and adjusted on the first training corpus to obtain a domain problem extraction model used for extracting domain problems from text corresponding to the target professional field.
3. The method according to claim 2, characterized in that The step of constructing a generative semantic model for extracting summary information from the text corresponding to the target professional field based on the first preset semantic model includes: Constructing a second training corpus, wherein the second training corpus is corpus data consisting of sample texts and summary information of the sample texts in the target professional field; The first preset semantic model is trained and adjusted on the second training corpus to obtain a generative semantic model.
4. The method according to claim 3, characterized in that The constructing of the second training corpus includes: Obtain sample text from the target professional field; Among the multiple sentences included in the sample text, a preset text summary evaluation algorithm is used to calculate an evaluation score of the semantic similarity between each sentence and the text composed of the remaining sentences; Based on the ranking of the evaluation scores of the sentences, a preset number of target sentences are obtained from the sample text to form summary information corresponding to the sample text; The remaining sentences in the sample text except the target sentence form a target text paragraph; The second training corpus is constructed according to the summary information corresponding to each of the target text paragraphs.
5. The method according to claim 1, characterized in that Before processing the target text and the domain question using the pre-built domain answer extraction model to output the domain answer corresponding to the domain question, the method further includes: Acquire a second preset semantic model, where the second preset semantic model is a semantic model obtained by training based on general corpus data; Constructing a third training corpus, wherein the third training corpus at least includes: sample texts in the target professional field, sample questions and sample answers corresponding to the sample texts, and the sample texts and the sample questions serve as input data for model training, and the sample answers serve as output data for model training; The second preset semantic model is trained and adjusted on the third training corpus to obtain a domain answer extraction model for extracting domain answers from text corresponding to the target professional field.
6. The method according to any one of claims 1 to 5, characterized in that The domain question and the corresponding domain answer are used to form a domain question-answer pair in the target professional domain, including: Establishing a mapping relationship between the domain question and the domain answer according to the semantic association relationship between the domain question and the domain answer; According to the mapping relationship, a domain question and answer pair is formed based on the domain question and the domain answer corresponding to the domain question.
7. The method according to claim 6, characterized in that After forming the domain question-answer pair, the method further includes: For each domain answer in the domain question-answer pair, based on the location of the domain answer in the target text, obtaining a target sentence containing the domain answer; The content information contained in the target sentence is used to expand the content information of the domain question-answer pair.
8. A device for automatically extracting professional field question-answer pairs, characterized in that: The device comprises: A determination unit, used for determining a target text to be processed from a target professional field; A first processing unit is used to process the target text using a pre-built domain problem extraction model, and output the domain problem corresponding to the target text; A second processing unit is used to process the target text and the domain question using a pre-built domain answer extraction model, and output a domain answer corresponding to the domain question; A composition unit is used to compose a domain question-answer pair in the target professional domain based on the domain question and the domain answer.
9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method for automatically extracting professional field question and answer pairs as described in any one of claims 1-7.
10. An electronic device, characterized in that: The device includes at least one processor, and at least one memory and a bus connected to the processor; Wherein, the processor and the memory communicate with each other through the bus; The processor is used to call the program instructions in the memory to execute the method for automatically extracting professional field question and answer pairs as described in any one of claims 1-7.
Citation Information
Cited By
Question and answer knowledge base construction method and system and storage medium
CN122309685A
