Text element extraction method based on natural language processing

Abstract

The invention relates to the technical field of text element extraction, in particular to a natural language processing-based text element extraction method, which comprises the following steps of: preprocessing an original text; tagging each vocabulary in the text as the part of speech by using a part-of-speech tagging tool; performing named entity recognition; key words with important significance are identified from the text by using a key word extraction technology; performing syntactic analysis to understand the relationship between words in the sentence and the structure of the sentence; using an emotion analysis technology to identify emotion and attitude expressed in the text; recognizing a theme in the text by applying a theme modeling technology; extracting a relationship between entities in the text; and generating the abstract of the text according to the keyword and sentence importance standard. According to the method, the element with the highest comprehensive score can be quickly calculated through comprehensive processing of the text, so that the extraction efficiency is higher, the extraction time is saved, and better use prospects and commercial values are brought.

Description

Technical Field

[0001] The present invention belongs to the technical field of text element extraction, and specifically relates to a text element extraction method based on natural language processing. Background Art

[0002] Text element extraction refers to extracting important parts or elements with specific information or features from a given text. These text elements can be words, phrases, sentences or paragraphs. They represent key information in the text or parts with specific meanings. The goal of text element extraction is to filter out the most relevant and informative parts from the text for further analysis, summary or application. Text element extraction has a wide range of applications in natural language processing, information retrieval, summary generation, intelligence analysis and other fields. By effectively extracting text elements, it can help users quickly obtain and understand the key content in a large amount of text information, thereby supporting more efficient decision-making and information processing processes.

[0003] Existing methods for extracting text elements are too process-oriented and not comprehensive enough, and cannot comprehensively process all characteristics, resulting in low extraction efficiency and long extraction time. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a text element extraction method based on natural language processing to solve the above technical problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a text element extraction method based on natural language processing, the element extraction method comprising:

[0006] S1: Preprocess the original text;

[0007] S2: Based on the preprocessed text, use the part-of-speech tagging tool to tag each word in the text with its part of speech;

[0008] S3: Perform named entity recognition to identify and classify named entities in text;

[0009] S4: Use keyword extraction techniques to identify key content representatives from the text;

[0010] S5: Based on syntactic analysis, it provides accurate sentence structure and the relationship between words;

[0011] S6: Use sentiment analysis techniques to identify the emotions and attitudes expressed in the text;

[0012] S7: Apply topic modeling techniques to identify topics in the text;

[0013] S8: Extract the relationship between entities in the text;

[0014] S9: Generate a summary of the text based on the criteria of keyword and sentence importance, extract the key information and elements of the text, and generate a general content summary.

[0015] Preferably, the step of S1 is:

[0016] Remove special characters from text, including punctuation marks, HTML tags and non-text symbols;

[0017] Split the text into sequences of words, removing stop words;

[0018] Stemming and morphological merging of words, converting different forms of words into their basic forms;

[0019] Convert all words in a text to the same format;

[0020] Use spell checker and dictionary to check for spelling mistakes in words and correct them.

[0021] Preferably, the step of S2 is:

[0022] Load the text that needs to be POS tagged into the environment of the POS tagging tool and input the text into the tool's interface;

[0023] When the part-of-speech tagging tool does not include word segmentation, first perform word segmentation on the text to divide the text into sequences of words and vocabulary;

[0024] Call the part-of-speech tagging function and method of the part-of-speech tagging tool to tag each word after word segmentation with its corresponding part-of-speech, analyze the grammatical role of each word in the context, and assign a part-of-speech tag;

[0025] Process and save the results returned by the part-of-speech tagging tool;

[0026] Verify the tagging results to ensure the accuracy and consistency of part-of-speech tagging.

[0027] Preferably, the step of S3 is:

[0028] Obtain and prepare training data containing annotated named entities;

[0029] Establish NER model based on rule-based method;

[0030] Use the trained NER model to perform entity recognition and classification on the text according to the part of speech of each word. The model outputs each recognized entity and its category.

[0031] Post-process the output of the NER model, including adjusting entity boundaries and resolving entity overlap issues to ensure that the final recognition results are accurate and consistent;

[0032] Use the standard evaluation metric precision to evaluate and optimize the performance of the NER model;

[0033] The precision evaluation formula is:

[0034]

[0035] Among them, TP is the true positive example, that is, the number of samples predicted by the model as positive examples and actually are positive examples, FP is the false positive example, that is, the number of samples predicted by the model as positive examples but actually are negative examples, and P r is the precision;

[0036] Integrate the trained NER model into practical applications.

[0037] Preferably, the step of S4 is:

[0038] Perform word frequency statistics on the text after word segmentation and record the frequency of each word appearing in the text;

[0039] Use keyword weight calculation method to measure the importance of each word in the text collection;

[0040] Sort the words according to the weight of the keywords;

[0041] According to the needs and application scenarios, select the top-ranked keywords as the key content representatives of the text;

[0042] The method for calculating the weight of keywords includes:

[0043] Calculate the frequency of occurrence of a keyword in a specific document; the formula is:

[0044]

[0045] In the formula, f is the number of times keyword t appears in text d, n is the total number of words in text d, and TF is the frequency of occurrence;

[0046]

[0047] Where D is the total number of texts in the document collection, |D T | is the number of texts containing keyword T, and IDF is the inverse document frequency;

[0048] Combine TF and IDF to calculate the weight of TF-IDF;

[0049] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D)

[0050] Where TF-IDF(t,d,D) is the weight of TF-IDF.

[0051] 6. The method for extracting text elements based on natural language processing according to claim 1, wherein the step S5 is:

[0052] Create a vocabulary graph and build a syntax tree using a syntactic analysis algorithm;

[0053] Mark the dependency relationship between each pair of words;

[0054] Use the syntax tree to determine the structure of the sentence, including phrase structure and syntactic hierarchy.

[0055] Preferably, the step of S6 is:

[0056] Use supervised learning methods to train sentiment classifiers to learn the relationship between different words and sentence structures in the text and sentiment categories;

[0057] For a given text, use the trained sentiment classifier to analyze the text into positive, negative and neutral sentiment categories;

[0058] Interpret the results of sentiment analysis, understand the emotions and attitudes expressed in the text, evaluate the accuracy and effectiveness of sentiment analysis, and verify and adjust the model through manually annotated datasets.

[0059] Preferably, the step of S7 is:

[0060] Apply the frequency and weight of words in the document, select a topic modeling algorithm, use the selected topic model to train the text-word frequency matrix, and then learn the topics and the weights of words in the topics from the data;

[0061] For new text data and previously unseen text, use the trained topic model to perform inference, map the text to the topic space, and determine the distribution of topics in each text;

[0062] Interpret the results of the topic model.

[0063] Preferably, the step of S8 is:

[0064] Perform named entity recognition on text;

[0065] Link the identified entities with entities in the knowledge base to ensure that each entity has a unique identifier. This step helps to disambiguate entities and ensure that the same entity mentioned in different texts is correctly associated.

[0066] Perform syntactic analysis on the text to capture the grammatical relationships between entities in the sentence;

[0067] Use relation extraction technology to extract the relationship between entities from the results of syntactic analysis;

[0068] Define rules to extract relations based on grammatical structure and lexical patterns;

[0069] Use neural network models to learn relationships between entities directly from text;

[0070] Classify and verify the extracted relationships;

[0071] Store the extracted relationships in the database and knowledge graph;

[0072] Evaluate the relation extraction system.

[0073] Preferably, the step of S9 is:

[0074] Assign importance rating to each sentence;

[0075] Position weighting: sentences near the beginning and end of the text are more important;

[0076] Sentence length: Shorter sentences are easier to understand and summarize;

[0077] Keyword frequency: sentences containing more keywords are more important;

[0078] Grammatical structure and connectives: Sentences with complex grammatical structures and connecting different paragraphs of information are more important;

[0079] Text coherence: retain sentences that maintain text coherence and logical flow;

[0080] The importance scoring formula is:

[0081] E C =β 1 ·X 1 +β 2 ·X 2 +…+β N ·X N

[0082] Among them, E C is the importance score, β 1 , β 2 ...β N is the structural feature of the text, X 1 , X 2 ...X N is the weight corresponding to the structural characteristics of the text;

[0083] Sort and filter sentences according to their importance scores, and select the most relevant and informative important sentences for generating summaries;

[0084] Based on the selected important sentences, connect them to form a coherent text summary in the order in which they appear in the original text;

[0085] Evaluate the generated summaries;

[0086] The generated text summary is output into a usable format and applied to relevant application scenarios such as information retrieval, document archiving, and content summary display.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] Through comprehensive processing of the text, this application can quickly calculate the elements with the highest comprehensive scores, thereby making the extraction efficiency higher, saving extraction time, and bringing better usage prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a diagram of the operation steps of the cloud system of the present invention. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0091] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0092] like Figure 1 As shown, the present invention provides a technical solution: a text element extraction method based on natural language processing, the element extraction method comprises:

[0093] S1: Preprocess the original text;

[0094] S2: Based on the preprocessed text, use the part-of-speech tagging tool to tag each word in the text with its part of speech;

[0095] S3: Perform named entity recognition to identify and classify named entities in text;

[0096] S4: Use keyword extraction techniques to identify key content representatives from the text;

[0097] S5: Based on syntactic analysis, it provides accurate sentence structure and the relationship between words;

[0098] S6: Use sentiment analysis techniques to identify the emotions and attitudes expressed in the text;

[0099] S7: Apply topic modeling techniques to identify topics in the text;

[0100] S8: Extract the relationship between entities in the text;

[0101] S9: Generate a summary of the text based on the criteria of keyword and sentence importance, extract the key information and elements of the text, and generate a general content summary.

[0102] Furthermore, the steps of S1 are:

[0103] Remove special characters from text, including punctuation marks, HTML tags and non-text symbols;

[0104] Split the text into sequences of words, removing stop words;

[0105] Stemming and morphological merging of words, converting different forms of words into their basic forms;

[0106] Convert all words in a text to the same format;

[0107] Use spell checker and dictionary to check for spelling mistakes in words and correct them.

[0108] Furthermore, the steps of S2 are:

[0109] Load the text that needs to be POS tagged into the environment of the POS tagging tool and input the text into the tool's interface;

[0110] When the part-of-speech tagging tool does not include word segmentation, first perform word segmentation on the text to divide the text into sequences of words and vocabulary;

[0111] Call the part-of-speech tagging function and method of the part-of-speech tagging tool to tag each word after segmentation with its corresponding part of speech, analyze the grammatical role of each word in the context, and assign a part-of-speech tag;

[0112] Process and save the results returned by the part-of-speech tagging tool;

[0113] Verify the tagging results to ensure the accuracy and consistency of part-of-speech tagging.

[0114] Furthermore, the steps of S3 are:

[0115] Obtain and prepare training data containing annotated named entities;

[0116] Establish NER model based on rule-based method;

[0117] Use the trained NER model to perform entity recognition and classification on the text according to the part of speech of each word. The model outputs each recognized entity and its category.

[0118] Post-process the output of the NER model, including adjusting entity boundaries and resolving entity overlap issues to ensure that the final recognition results are accurate and consistent;

[0119] Use the standard evaluation metric precision to evaluate and optimize the performance of the NER model;

[0120] The precision evaluation formula is:

[0121]

[0122] Among them, TP is the true positive example, that is, the number of samples predicted by the model as positive examples and actually are positive examples, FP is the false positive example, that is, the number of samples predicted by the model as positive examples but actually are negative examples, and P r is the precision;

[0123] Integrate the trained NER model into practical applications.

[0124] Furthermore, the steps of S4 are:

[0125] Perform word frequency statistics on the text after word segmentation and record the frequency of each word appearing in the text;

[0126] Use keyword weight calculation method to measure the importance of each word in the text collection;

[0127] Sort the words according to the weight of the keywords;

[0128] According to the needs and application scenarios, select the top-ranked keywords as the key content representatives of the text;

[0129] The method for calculating the weight of keywords includes:

[0130] Calculate the frequency of occurrence of a keyword in a specific document; the formula is:

[0131]

[0132] In the formula, f is the number of times keyword t appears in text d, n is the total number of words in text d, and TF is the frequency of occurrence;

[0133]

[0134] Where D is the total number of texts in the document collection, |D T | is the number of texts containing keyword T, and IDF is the inverse document frequency;

[0135] Combine TF and IDF to calculate the weight of TF-IDF;

[0136] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D)

[0137] Where TF-IDF(t,d,D) is the weight of TF-IDF.

[0138] Furthermore, the steps of S5 are:

[0139] Create a vocabulary graph and build a syntax tree using a syntactic analysis algorithm;

[0140] Mark the dependency relationship between each pair of words;

[0141] Use the syntax tree to determine the structure of the sentence, including phrase structure and syntactic hierarchy.

[0142] Furthermore, the steps of S6 are:

[0143] Use supervised learning methods to train sentiment classifiers to learn the relationship between different words and sentence structures in the text and sentiment categories;

[0144] For a given text, use the trained sentiment classifier to analyze the text into positive, negative and neutral sentiment categories;

[0145] Interpret the results of sentiment analysis, understand the emotions and attitudes expressed in the text, evaluate the accuracy and effectiveness of sentiment analysis, and verify and adjust the model through manually annotated datasets.

[0146] Furthermore, the steps of S7 are:

[0147] Apply the frequency and weight of words in the document, select a topic modeling algorithm, use the selected topic model to train the document-word frequency matrix, and then learn the topics and the weights of words in the topics from the data;

[0148] For new text data and previously unseen text, use the trained topic model to perform inference, map the text to the topic space, and determine the distribution of topics in each text;

[0149] Interpret the results of the topic model.

[0150] Furthermore, the steps of S8 are:

[0151] Perform named entity recognition on text;

[0152] Link the identified entities with entities in the knowledge base to ensure that each entity has a unique identifier. This step helps to disambiguate entities and ensure that the same entity mentioned in different texts is correctly associated.

[0153] Perform syntactic analysis on the text to capture the grammatical relationships between entities in the sentence;

[0154] Use relation extraction technology to extract the relationship between entities from the results of syntactic analysis;

[0155] Define rules to extract relations based on grammatical structure and lexical patterns;

[0156] Use neural network models to learn relationships between entities directly from text;

[0157] Classify and verify the extracted relationships;

[0158] Store the extracted relationships in the database and knowledge graph;

[0159] Evaluate the relation extraction system.

[0160] Preferably, the step of S9 is:

[0161] Assign importance rating to each sentence;

[0162] Position weighting: sentences near the beginning and end of the text are more important;

[0163] Sentence length: Shorter sentences are easier to understand and summarize;

[0164] Keyword frequency: sentences containing more keywords are more important;

[0165] Grammatical structure and connectors: Sentences with complex grammatical structures and connecting different paragraphs of information are more important;

[0166] Text coherence: retain sentences that maintain text coherence and logical flow;

[0167] The importance scoring formula is:

[0168] E C =β 1 ·X 1 +β 2 ·X 2 +…+β N ·X N

[0169] Among them, E C is the importance score, β 1 , β 2 ...β N is the structural feature of the text, X 1 , X 2 ...X N is the weight corresponding to the structural characteristics of the text;

[0170] Sort and filter sentences according to their importance scores, and select the most relevant and informative important sentences for generating summaries;

[0171] Based on the selected important sentences, connect them to form a coherent text summary in the order in which they appear in the original text;

[0172] Evaluate the generated summaries;

[0173] The generated text summary is output into a usable format and applied to relevant application scenarios such as information retrieval, document archiving, and content summary display.

[0174] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0175] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text element extraction method based on natural language processing, characterized in that: include: S1: Preprocess the original text; S2: Based on the preprocessed text, use the part-of-speech tagging tool to tag each word in the text with its part of speech; S3: Perform named entity recognition to identify and classify named entities in text; S4: Use keyword extraction techniques to identify key content representatives from the text; S5: Based on syntactic analysis, it provides accurate sentence structure and the relationship between words; S6: Use sentiment analysis techniques to identify the emotions and attitudes expressed in the text; S7: Apply topic modeling techniques to identify topics in the text; S8: Extract the relationship between entities in the text; S9: Generate a summary of the text based on the criteria of keyword and sentence importance, extract the key information and elements of the text, and generate a general content summary.

2. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S1 are: Remove special characters from text, including punctuation marks, HTML tags and non-text symbols; Split the text into sequences of words, removing stop words; Stemming and morphological merging of words, converting different forms of words into their basic forms; Convert all words in a text to the same format; Use spell checker and dictionary to check for spelling mistakes in words and correct them.

3. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S2 are: Load the text that needs to be POS tagged into the environment of the POS tagging tool and input the text into the tool's interface; When the part-of-speech tagging tool does not include word segmentation, first perform word segmentation on the text to divide the text into sequences of words and vocabulary; Call the part-of-speech tagging function and method of the part-of-speech tagging tool to tag each word after word segmentation with its corresponding part-of-speech, analyze the grammatical role of each word in the context, and assign a part-of-speech tag; Process and save the results returned by the part-of-speech tagging tool; Verify the tagging results to ensure the accuracy and consistency of part-of-speech tagging.

4. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S3 are: Obtain and prepare training data containing annotated named entities; Establish NER model based on rule-based method; Use the trained NER model to perform entity recognition and classification on the text according to the part of speech of each word. The model outputs each recognized entity and its category. Post-process the output of the NER model, including adjusting entity boundaries and resolving entity overlap issues to ensure that the final recognition results are accurate and consistent; Use the standard evaluation metric precision to evaluate and optimize the performance of the NER model; The precision evaluation formula is: Among them, TP is the true positive example, that is, the number of samples predicted by the model as positive examples and actually are positive examples, FP is the false positive example, that is, the number of samples predicted by the model as positive examples but actually are negative examples, and P r is the precision; Integrate the trained NER model into practical applications.

5. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S4 are: Perform word frequency statistics on the text after word segmentation and record the frequency of each word appearing in the text; Use keyword weight calculation method to measure the importance of each word in the text collection; Sort the words according to the weight of the keywords; According to the needs and application scenarios, select the top-ranked keywords as the key content representatives of the text; The method for calculating the weight of keywords includes: Calculate the frequency of occurrence of a keyword in a specific document; the formula is: In the formula, f is the number of times keyword t appears in text d, n is the total number of words in text d, and TF is the frequency of occurrence; Where D is the total number of texts in the document collection, |D T | is the number of texts containing keyword T, and IDF is the inverse document frequency; Combine TF and IDF to calculate the weight of TF-IDF; TF-IDF(t,d,D)=TF(t,d)×IDF(t,D) Where TF-IDF(t,d,D) is the weight of TF-IDF.

6. The method for extracting text elements based on natural language processing according to claim 1, characterized in that: The steps of S5 are: Create a vocabulary graph and build a syntax tree using a syntactic analysis algorithm; Mark the dependency relationship between each pair of words; Use the syntax tree to determine the structure of the sentence, including phrase structure and syntactic hierarchy.

7. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S6 are: Use supervised learning methods to train sentiment classifiers to learn the relationship between different words and sentence structures in the text and sentiment categories; For a given text, use the trained sentiment classifier to analyze the text into positive, negative and neutral sentiment categories; Interpret the results of sentiment analysis, understand the emotions and attitudes expressed in the text, evaluate the accuracy and effectiveness of sentiment analysis, and verify and adjust the model through manually annotated datasets.

8. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S7 are: Apply the frequency and weight of words in the document, select a topic modeling algorithm, use the selected topic model to train the text-word frequency matrix, and then learn the topics and the weights of words in the topics from the data; For the text of the new text data, the trained topic model is used for inference, the text is mapped to the topic space, and the distribution of topics in each text is determined.

9. The text element extraction method based on natural language processing according to claim 1 is characterized in that: The steps of S8 are: Perform named entity recognition on text; Link the identified entities with entities in the knowledge base; Perform syntactic analysis on the text to capture the grammatical relationships between entities in the sentence; Use relation extraction technology to extract the relationship between entities from the results of syntactic analysis; Define rules to extract relations based on grammatical structure and lexical patterns; Use neural network models to learn relationships between entities directly from text; Evaluate the relation extraction system.

10. The text element extraction method based on natural language processing according to claim 1, characterized in that: The steps of S9 are: Assign importance rating to each sentence; The importance scoring formula is: E C =β1·X1+β2·X2+…+β N ·X N Among them, E C is the importance score, β1, β2, ... β N It is the structural feature of the text, X1, X2...X N is the weight corresponding to the structural characteristics of the text; Sort and filter sentences according to their importance scores, and select the most relevant and informative important sentences for generating summaries; Based on the selected said important sentences, they are connected to form a coherent text summary in the order in which they appear in the original text.

Images