Key content extraction method

Abstract

The present invention discloses a key content extraction method, which obtains subject information of key content to be extracted, generates a corresponding subject knowledge base according to the subject information; extracts original text from the subject knowledge base, performs data processing on the original text, and obtains corresponding target text; performs word segmentation processing and cluster analysis on the target text, and obtains key content in the target text according to a preset analysis method, wherein the key content includes knowledge points and / or keywords. The technical solution achieves the purpose of automatically extracting key content from the text corresponding to the subject, improves the extraction efficiency and extraction accuracy of key content, and compared with the method of manually extracting by manually marking exercises, the key content extraction method of the present invention improves work efficiency and also saves a lot of manpower.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a key content extraction method. Background Art

[0002] With the continuous development and progress of computer technology and Internet technology, as well as the gradual popularization of intelligent electronic products, students are gradually beginning to learn with the help of electronic products, considering the intelligence and convenience of electronic product learning. Therefore, in the teaching process of students, there are also a large number of electronic exercises. At present, such exercises are basically manually marked to confirm the knowledge points and corresponding keywords, which has low work efficiency and large workload. Summary of the invention

[0003] The invention provides a key content extraction method, aiming to realize automatic extraction of key content corresponding to electronic exercises.

[0004] The present invention provides a key content extraction method, the method comprising:

[0005] Obtaining subject information of key content to be extracted, and generating a corresponding subject knowledge base based on the subject information;

[0006] Extracting original text from the subject knowledge base, performing data processing on the original text, and obtaining corresponding target text;

[0007] The target text is subjected to word segmentation processing and cluster analysis, and key contents in the target text are obtained according to a preset analysis method, wherein the key contents include knowledge points and / or keywords.

[0008] Furthermore, the step of obtaining subject information of key content to be extracted and generating a corresponding subject knowledge base according to the subject information includes:

[0009] Parsing the subject information to obtain known subject knowledge points and known subject keywords corresponding to the subject information;

[0010] Based on the acquired known subject knowledge points and known subject keywords, a subject knowledge base including the known subject knowledge points and known subject keywords is generated.

[0011] Furthermore, generating a subject knowledge base including the known subject knowledge points and the known subject keywords based on the acquired known subject knowledge points and the known subject keywords includes:

[0012] According to the acquired known subject knowledge points and known subject keywords, the known subject knowledge points and known subject keywords are annotated, and the annotated known subject knowledge points and known subject keywords are used as label samples to generate a subject knowledge base containing the label samples.

[0013] Furthermore, the obtaining of subject information of key contents to be extracted and generating a corresponding subject knowledge base according to the subject information include:

[0014] Obtaining subject information of key content to be extracted, parsing the subject information, and obtaining subject types and subject characteristics corresponding to the subject information;

[0015] According to the subject type and subject characteristics, obtaining subject vocabulary and high-frequency vocabulary of the profession corresponding to the subject type and subject characteristics;

[0016] The subject vocabulary and high-frequency vocabulary are annotated, and the annotated subject vocabulary and high-frequency vocabulary are used as label samples to generate a subject knowledge base containing the label samples.

[0017] Furthermore, the obtaining of subject information of key contents to be extracted and generating a corresponding subject knowledge base according to the subject information include:

[0018] Obtaining subject information of key content to be extracted, and collecting known subject knowledge points and known subject keywords from the subject information;

[0019] Based on the collected known subject knowledge points and known subject keywords, a subject knowledge base corresponding to the subject knowledge graph is generated.

[0020] Furthermore, the data processing of the original text to obtain the corresponding target text includes:

[0021] According to the subject knowledge base, data preprocessing is performed on the original text, and irrelevant characters including spaces in the original text are removed to obtain a corresponding target text.

[0022] Furthermore, the word segmentation and cluster analysis of the target text and obtaining the key content of the target text according to a preset analysis method include:

[0023] Perform word segmentation processing on the target text to obtain a number of corresponding word segmentation words, and calculate the current popularity value of each word segmentation word;

[0024] Performing cluster analysis on the plurality of segmentation words to obtain segmentation word sets corresponding to each type of segmentation word;

[0025] Extracting target words from each segmentation vocabulary set according to the preset N vocabulary extraction methods to obtain a plurality of extracted vocabulary sets corresponding to each segmentation vocabulary set, each extracted vocabulary set including the corresponding target words;

[0026] Determine the comprehensive effective value corresponding to each extracted vocabulary set according to the current popularity value of the target vocabulary;

[0027] Sort the comprehensive effective values ​​from large to small to obtain a set of extracted words ranked in the top n;

[0028] The key content of each extracted vocabulary set in the first n extracted vocabulary sets is extracted to obtain the key content in the target text.

[0029] Furthermore, the calculation of the current heat value of each word segmentation vocabulary includes:

[0030] Using formula (1), calculate the current heat value of each segmentation word:

[0031]

[0032] In formula (1), S k Indicates the current heat value of the kth word segmentation vocabulary; β k represents the lexical attribute value of the kth segmentation word, which is a preset value in the range of [1,5]; n represents the number of unit time periods included in the preset total time period; χ ki represents the attention degree of the k-th word segment in the i-th unit time period; k ′ represents the average attention of the k-th word segment in the total time period; kmax represents the maximum attention of the k-th word segment in all unit time periods in the total time period;

[0033] Among them, the x ki The calculation method is as follows:

[0034]

[0035] Among them, p ki P1 represents the search frequency of the k-th word segmentation term in the i-th unit time period; i Represents the total frequency of searching for different word segments in the i-th unit time period.

[0036] Furthermore, the step of determining the comprehensive effective value corresponding to each extracted vocabulary set according to the current popularity value of the target vocabulary includes:

[0037] Using formula (3) and formula (4), we can calculate the comprehensive effective value corresponding to each extracted vocabulary set, and we have:

[0038]

[0039]

[0040] Among them, Z a represents the comprehensive effective value of the ath extracted vocabulary set; m represents the total number of all words finally extracted when the target words in the ath extracted vocabulary set are extracted using the N vocabulary extraction methods; S aj represents the current heat value of the jth word extracted; p amax represents the probability of being extracted when the target words in the a-th extracted word set are extracted using the N word extraction methods respectively, and the word that is extracted the most times among the extracted words; p amin represents the extraction probability corresponding to the word with the least number of extractions among the extracted words when the target words in the a-th extracted word set are extracted respectively by using the N kinds of word extraction methods;

[0041] d aj represents the total number of times the jth word extracted appears in the extraction process using N word extraction methods; k ad It represents the number of words extracted in the process of extracting the a-th extracted word set by using the d-th word extraction method.

[0042] The key content extraction method of the present invention obtains the subject information of the key content to be extracted, and generates a corresponding subject knowledge base according to the subject information; extracts the original text from the subject knowledge base, performs data processing on the original text, and obtains the corresponding target text; performs word segmentation processing and cluster analysis on the target text, and obtains the key content in the target text according to a preset analysis method, and the key content includes knowledge points and / or keywords; the purpose of automatically extracting the knowledge points and keywords corresponding to the electronic exercises is achieved, and the extraction efficiency and extraction accuracy of the knowledge points and keywords are improved. Compared with the method of manually extracting by manually annotating exercises, the knowledge point and keyword extraction method of the present invention improves work efficiency, reduces the error rate, and also saves a lot of manpower.

[0043] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the contents pointed out in the written description, claims, and drawings.

[0044] The technical solution of the present invention is further described below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0046] Figure 1 It is a schematic diagram of the workflow of an implementation method of the key content extraction method of the present invention. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0048] The present invention provides a key content extraction method, which solves the problems of low efficiency and heavy workload in manually annotating exercises, and achieves the purpose of automatically extracting knowledge points and keywords corresponding to electronic exercises.

[0049] like Figure 1 As shown, Figure 1 It is a schematic diagram of the workflow of an implementation of the key content extraction method of the present invention; the key content extraction method of the present invention can be implemented as steps S10-S30 described below.

[0050] Step S10: Obtain subject information of key content to be extracted, and generate a corresponding subject knowledge base based on the subject information.

[0051] In the embodiment of the present invention, the system obtains subject information of key content to be extracted; wherein the subject information includes: subjects corresponding to mathematics, Chinese, physics, chemistry, etc. and all electronic exercises corresponding to the subjects. According to the subject information, a corresponding subject knowledge base is generated; in order to facilitate the extraction of knowledge points and keywords, the subject knowledge base may only contain subject information corresponding to one subject.

[0052] Step S20: extracting the original text from the subject knowledge base, performing data processing on the original text, and obtaining the corresponding target text.

[0053] The original text can be electronic exercises or other subject-related texts.

[0054] Based on the generated subject knowledge base, the original text is extracted from the subject knowledge base. When the original text is subjected to data preprocessing, it is necessary to remove the content irrelevant to the words in the original text, such as spaces and other characters without any symbolic meaning, so as to obtain the corresponding target text.

[0055] Step S30: performing word segmentation processing and cluster analysis on the target text, and obtaining key content in the target text according to a preset analysis method, wherein the key content includes knowledge points and / or keywords.

[0056] In an embodiment of the present invention, the target text is segmented to obtain corresponding segmented vocabulary, and then the segmented vocabulary is clustered to obtain a corresponding segmented set. The preset analysis method includes but is not limited to: based on the obtained segmented set, according to the corresponding extraction method, target vocabulary is extracted from the segmented set to obtain a corresponding extraction set, and based on the extraction set, corresponding knowledge points and / or keywords are extracted and output.

[0057] In one embodiment, the acquisition of subject information of key content to be extracted and the generation of a corresponding subject knowledge base according to the subject information may be implemented as follows:

[0058] Parsing the subject information to obtain known subject knowledge points and known subject keywords corresponding to the subject information;

[0059] Based on the acquired known subject knowledge points and known subject keywords, a subject knowledge base including the known subject knowledge points and known subject keywords is generated.

[0060] In one embodiment, generating a subject knowledge base including the known subject knowledge points and the known subject keywords according to the acquired known subject knowledge points and the known subject keywords may be implemented as follows:

[0061] According to the acquired known subject knowledge points and known subject keywords, the known subject knowledge points and known subject keywords are annotated, and the annotated known subject knowledge points and known subject keywords are used as label samples to generate a subject knowledge base containing the label samples.

[0062] In one embodiment, the obtaining of subject information of key content to be extracted and the generation of a corresponding subject knowledge base according to the subject information may also be implemented in the following manner:

[0063] Obtaining subject information of key content to be extracted, parsing the subject information, and obtaining subject types and subject characteristics corresponding to the subject information;

[0064] According to the subject type and subject characteristics, obtaining subject vocabulary and high-frequency vocabulary of the profession corresponding to the subject type and subject characteristics;

[0065] The subject vocabulary and high-frequency vocabulary are annotated, and the annotated subject vocabulary and high-frequency vocabulary are used as label samples to generate a subject knowledge base containing the label samples.

[0066] In the embodiment of the present invention, known professional subject vocabulary and high-frequency vocabulary are annotated and stored as label samples in the corresponding subject knowledge base. For example, torque and Newton (an international unit for measuring force) that appear in physics are annotated and stored as corresponding label samples in the physics subject knowledge base.

[0067] In one embodiment, the obtaining of subject information of key content to be extracted and the generation of a corresponding subject knowledge base according to the subject information may also be implemented in the following manner:

[0068] Obtaining subject information of key content to be extracted, and collecting known subject knowledge points and known subject keywords from the subject information;

[0069] Based on the collected known subject knowledge points and known subject keywords, a subject knowledge base corresponding to the subject knowledge graph is generated.

[0070] In one embodiment, the data processing of the original text to obtain the corresponding target text can be implemented as follows:

[0071] According to the subject knowledge base, data preprocessing is performed on the original text, and irrelevant characters including spaces in the original text are removed to obtain a corresponding target text.

[0072] In one embodiment, the word segmentation and cluster analysis of the target text and obtaining the key content of the target text according to a preset analysis method can be implemented as follows:

[0073] Perform word segmentation processing on the target text to obtain a number of corresponding word segmentation words, and calculate the current popularity value of each word segmentation word;

[0074] Performing cluster analysis on the plurality of segmentation words to obtain segmentation word sets corresponding to each type of segmentation word;

[0075] Extracting target words from each segmentation vocabulary set according to the preset N vocabulary extraction methods to obtain a plurality of extracted vocabulary sets corresponding to each segmentation vocabulary set, each extracted vocabulary set including the corresponding target words;

[0076] Determine the comprehensive effective value corresponding to each extracted vocabulary set according to the current popularity value of the target vocabulary;

[0077] Sort the comprehensive effective values ​​from large to small to obtain a set of extracted words ranked in the top n;

[0078] The key content of each extracted vocabulary set in the first n extracted vocabulary sets is extracted to obtain the key content in the target text.

[0079] In one embodiment, the calculation of the current popularity value of each segmentation word can be implemented as follows:

[0080] Using formula (1), calculate the current heat value of each segmentation word:

[0081]

[0082] In formula (1), S k Indicates the current heat value of the kth word segmentation vocabulary; β k represents the lexical attribute value of the kth segmentation word, which is a preset value in the range of [1,5]; n represents the number of unit time periods included in the preset total time period; χ ki represents the attention degree of the k-th word segment in the i-th unit time period; k represents the average attention of the k-th word segment in the total time period; kmax represents the maximum attention of the k-th word segment in all unit time periods in the total time period;

[0083] Among them, the x ki The calculation method is as follows:

[0084]

[0085] Among them, p ki P1 represents the search frequency of the k-th word segmentation term in the i-th unit time period; i Represents the total frequency of searching for different word segments in the i-th unit time period.

[0086] In one embodiment, the determining of the comprehensive effective value corresponding to each extracted vocabulary set according to the current popularity value of the target vocabulary may be implemented as follows:

[0087] Using formula (3) and formula (4), we can calculate the comprehensive effective value corresponding to each extracted vocabulary set, and we have:

[0088]

[0089]

[0090] Among them, Z arepresents the comprehensive effective value of the ath extracted vocabulary set; m represents the total number of all words finally extracted when the target words in the ath extracted vocabulary set are extracted using the N vocabulary extraction methods; S aj represents the current heat value of the jth word extracted; p amax represents the probability of being extracted when the target words in the a-th extracted word set are extracted using the N word extraction methods respectively, and the word that is extracted the most times among the extracted words; p amin represents the extraction probability corresponding to the word with the least number of extractions among the extracted words when the target words in the a-th extracted word set are extracted respectively by using the N kinds of word extraction methods;

[0091] d aj represents the total number of times the jth word extracted appears in the extraction process using N word extraction methods; k ad It represents the number of words extracted in the process of extracting the a-th extracted word set by using the d-th word extraction method.

[0092] In the implementation of the present invention, the current popularity value of the target vocabulary is determined based on the current popularity value of the segmentation vocabulary, and the number of vocabulary types in the extraction set is less than or equal to the number of vocabulary types in the corresponding segmentation set.

[0093] The above-mentioned N-word extraction method may be performed by using attributes related to word popularity or by using attributes related to the difficulty level corresponding to the words.

[0094] In the implementation of the present invention, preprocessing is performed to obtain the target problem and improve the efficiency of subsequent word segmentation acquisition; the current popularity value of each word is calculated to provide a data basis for acquiring knowledge points; cluster analysis is performed on the words and a preset extraction method is used to facilitate the construction of an accurate extraction set, which can effectively and comprehensively screen key words, and calculate the comprehensive effective value of each extraction set to facilitate the determination of the effectiveness of the set, and by screening the first n1 extraction sets and based on the subject knowledge base, it is easy to obtain effective key content.

[0095] In one embodiment, Figure 1 In the embodiment, "step S10, obtaining subject information of key content to be extracted, and generating a corresponding subject knowledge base according to the subject information" can also be implemented by the following technical means:

[0096] Obtain subject information of key content to be extracted, and collect known subject knowledge points and known subject keywords from the subject information; generate a subject knowledge base corresponding to the subject knowledge graph based on the collected known subject knowledge points and known subject keywords.

[0097] In the embodiment of the present invention, the concept of knowledge graph is introduced, and the association relationship between different words in the subject information is displayed accordingly through the knowledge graph. This processing method is suitable for subjects that use graphs to represent the association relationship between different words, such as the description of chromosome-related information between multiple individuals with genetic relationships in biology. In the embodiment of the present invention, known subject knowledge points and known subject keywords can also be marked as label samples and stored in the subject knowledge base.

[0098] The key content extraction method of the present invention obtains the subject information of the key content to be extracted, generates a corresponding subject knowledge base according to the subject information; extracts the original text from the subject knowledge base, performs data processing on the original text, and obtains the corresponding target text; performs word segmentation processing and cluster analysis on the target text, and obtains the key content in the target text according to a preset analysis method, and the key content includes knowledge points and / or keywords. This technical solution can achieve the purpose of automatically extracting key content from the text corresponding to the subject, improves the extraction efficiency and extraction accuracy of key content, and compared with the method of manually extracting by manually marking exercises, the knowledge point and keyword extraction method of the present invention improves work efficiency, reduces error rate, and also saves a lot of manpower.

[0099] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A key content extraction method, characterized in that: The method comprises: Obtaining subject information of key content to be extracted, and generating a corresponding subject knowledge base based on the subject information; Extracting original text from the subject knowledge base, performing data processing on the original text, and obtaining corresponding target text; Performing word segmentation and cluster analysis on the target text, and obtaining key content in the target text according to a preset analysis method, wherein the key content includes knowledge points and / or keywords; The word segmentation and cluster analysis of the target text and obtaining the key content of the target text according to a preset analysis method include: Perform word segmentation processing on the target text to obtain a number of corresponding word segmentation words, and calculate the current popularity value of each word segmentation word; Performing cluster analysis on the plurality of segmentation words to obtain segmentation word sets corresponding to each type of segmentation word; Extracting target words from each segmentation vocabulary set according to the preset N vocabulary extraction methods to obtain a plurality of extracted vocabulary sets corresponding to each segmentation vocabulary set, each extracted vocabulary set including the corresponding target words; Determine the comprehensive effective value corresponding to each extracted vocabulary set according to the current popularity value of the target vocabulary; Sort the comprehensive effective values ​​from large to small to obtain a set of extracted words arranged in the first n; Extracting key content from each of the first n extracted vocabulary sets to obtain key content in the target text; The step of calculating the current heat value of each word segmentation vocabulary includes: Using formula (1), calculate the current heat value of each segmentation word: In formula (1), S k Indicates the current heat value of the kth word segmentation vocabulary; β k represents the lexical attribute value of the kth word segmentation vocabulary, which is a preset value and has a value range of [1,5]; n represents the number of unit time periods included in the preset total time period; χ ki represents the attention degree of the k-th word segment in the i-th unit time period; k ′ represents the average attention of the k-th word segment in the total time period; kmax represents the maximum attention of the k-th word segment in all unit time periods in the total time period; Among them, the x ki The calculation method is as follows: Among them, p ki P1 represents the search frequency of the k-th word segmentation term in the i-th unit time period; i Represents the total frequency of searching for different word segments in the i-th unit time period; The step of determining the comprehensive effective value corresponding to each extracted vocabulary set according to the current popularity value of the target vocabulary includes: Using formula (3) and formula (4), we can calculate the comprehensive effective value corresponding to each extracted vocabulary set, and we have: Among them, Z a represents the comprehensive effective value of the ath extracted vocabulary set; m represents the total number of all words finally extracted when the target words in the ath extracted vocabulary set are extracted using the N vocabulary extraction methods; S aj represents the current heat value of the jth word extracted; p amax represents the probability of being extracted when the target words in the a-th extracted word set are extracted using the N word extraction methods respectively, and the word that is extracted the most times among the extracted words; p amin represents the extraction probability corresponding to the word with the least number of extractions among the extracted words when the target words in the a-th extracted word set are extracted respectively by using the N kinds of word extraction methods; d aj represents the total number of times the jth word extracted appears in the extraction process using N word extraction methods; k ad It represents the number of words extracted in the process of extracting the a-th extracted word set by using the d-th word extraction method.

2. The key content extraction method as claimed in claim 1, characterized in that: The step of obtaining subject information of key content to be extracted and generating a corresponding subject knowledge base according to the subject information includes: Parsing the subject information to obtain known subject knowledge points and known subject keywords corresponding to the subject information; Based on the acquired known subject knowledge points and known subject keywords, a subject knowledge base including the known subject knowledge points and known subject keywords is generated.

3. The key content extraction method as claimed in claim 2, characterized in that: The step of generating a subject knowledge base including the known subject knowledge points and the known subject keywords according to the acquired known subject knowledge points and the known subject keywords comprises: According to the acquired known subject knowledge points and known subject keywords, the known subject knowledge points and known subject keywords are annotated, and the annotated known subject knowledge points and known subject keywords are used as label samples to generate a subject knowledge base containing the label samples.

4. The key content extraction method as claimed in claim 1, characterized in that: The step of obtaining subject information of key content to be extracted and generating a corresponding subject knowledge base according to the subject information includes: Obtaining subject information of key content to be extracted, parsing the subject information, and obtaining subject types and subject characteristics corresponding to the subject information; According to the subject type and subject characteristics, obtaining subject vocabulary and high-frequency vocabulary of the profession corresponding to the subject type and subject characteristics; The subject vocabulary and high-frequency vocabulary are annotated, and the annotated subject vocabulary and high-frequency vocabulary are used as label samples to generate a subject knowledge base containing the label samples.

5. The key content extraction method as claimed in claim 1, characterized in that: The step of obtaining subject information of key content to be extracted and generating a corresponding subject knowledge base according to the subject information includes: Obtaining subject information of key content to be extracted, and collecting known subject knowledge points and known subject keywords from the subject information; Based on the collected known subject knowledge points and known subject keywords, a subject knowledge base corresponding to the subject knowledge graph is generated.

6. The key content extraction method according to any one of claims 1 to 5, characterized in that: The data processing of the original text to obtain the corresponding target text includes: According to the subject knowledge base, data preprocessing is performed on the original text, and irrelevant characters including spaces in the original text are removed to obtain a corresponding target text.

Images