A text content analysis method and system combining theme and context

Abstract

The present invention relates to the technical field of text data analysis, and discloses a text content analysis method and system combining themes and contexts, including a text content decomposition module, an external data acquisition module, a text analysis module, and a text content evaluation module. The text content decomposition module includes a sentence decomposition unit, a word decomposition unit, and a paragraph decomposition unit. The text analysis module includes a sentence analysis unit, a paragraph analysis unit, and a word analysis unit. The system decomposes the input text four times independently according to sentences, paragraphs, and words, and obtains a sentence data set, a paragraph data set, a noun data set, and a sentiment word data set. Based on the data sets, a first similarity factor set, a second similarity factor set, and a third similarity factor set are obtained. The first similarity factor set, the second similarity factor set, and the third similarity factor set are comprehensively evaluated to output topic keywords, and the text is classified, thereby ensuring the accuracy of keyword extraction in the text.

Description

Technical Field

[0001] The present invention relates to the technical field of text data analysis, and in particular to a text content analysis method and system combining theme and context. Background Art

[0002] With the rapid development of the internet and the widespread adoption of personal computers, more and more information and data are being distributed in the form of electronic documents via the Hypertext Transfer Protocol. This has led to significant challenges in the speed and ability of data retrieval. Accurately and promptly retrieving the information users need from this vast world of information has been a major challenge in the development of the internet.

[0003] Search engine technology is a relatively mature information retrieval technique. However, with the rapid growth of internet data, search engine shortcomings are gradually becoming apparent. Traditional search engines typically only accept keywords as input. For ordinary users, it is often difficult to condense a small number of keywords to accurately express their query intent. Furthermore, search engine results are not concise and accurate answers, but rather a list of web page fragments. These web page fragments often contain a large amount of noise data, and users still need to read these web page fragments and even the corresponding original web pages to find the answers they need. However, when users upload text, it is easy for the keywords in the text subject to deviate from the actual keywords in the text, resulting in keyword classification errors, which in turn leads to low text retrieval efficiency. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the deficiencies of the existing technology, the present invention provides a text content analysis method and system that combines theme and context, which has the advantages of accurately extracting keywords from text and solves the above-mentioned technical problems.

[0006] (2) Technical solution

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a text content analysis system combining theme and context, comprising a text content disassembly module, an external data acquisition module, a text analysis module and a text content evaluation module; the text content disassembly module comprises a sentence disassembly unit, a word disassembly unit and a paragraph disassembly unit, the sentence disassembly unit is used to read the input text, identify sentence delimiters, and decompose the text into several sentences according to the sentence delimiters, and construct a sentence data set, the word deassembly unit is used to decompose the words in each sentence in the sentence data set, and classify the words according to part of speech to obtain a noun data set and a sentiment word data set, the paragraph disassembly unit is used to read the paragraph delimiters in the text to decompose the text into several paragraphs, and construct a paragraph data set; the external data acquisition module is used to collect all text within 3 days of the current browsing page content, and outputs the subject label of each text to constitute an external data set, wherein the subject label of the text stores the classification mark and text keywords of the text; the text analysis module includes a sentence analysis unit, a paragraph analysis unit and a word analysis unit, the sentence analysis unit extracts a number of sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs a first similarity factor set; the paragraph analysis unit extracts a number of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs a second similarity factor set; the word analysis unit compares the noun data set, the sentiment word data set and the external data set, and comprehensively outputs a third similarity factor set; the text content evaluation module outputs the subject keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set and the third similarity factor set, and classifies the text.

[0008] As a preferred technical solution of the present invention, the sentence decomposition unit reads the input text, identifies sentence delimiters, decomposes the text into several sentences based on the sentence delimiters, and constructs a sentence data set. The specific expression is as follows: in, Represents a statement dataset, Respectively represent the first statement formed by the statement delimiter, , the first statements, , the first statements.

[0009] As a preferred technical solution of the present invention, the word decomposition unit decomposes the words in each sentence in the sentence data set and classifies the words according to the part of speech to obtain the noun data set and the sentiment word data set. The specific steps are as follows: Step A1: statements Remove the stop words in statements Several words; Step A2: statements The nouns in several words are classified into the noun dataset, and the sentiment words are classified into the sentiment word dataset until the common The division is completed by the following statement: in, represents a noun dataset, Respectively express the The first noun data obtained by dividing the sentences, , No. Noun data, , No. Noun data, and the Noun data Store nouns and the frequency of noun occurrence. represents the sentiment word dataset, Respectively express the The first sentiment word data obtained by dividing the sentences, , No. Sentiment word data, , No. sentiment word data, and the Sentiment word data The number of characters between the nearest noun and the sentiment level of the sentiment word is stored. The sentiment level of the sentiment word is directly obtained by the natural language processing toolkit NLTK.

[0010] As a preferred technical solution of the present invention, the paragraph decomposition unit reads the paragraph separators in the text to decompose the text into several paragraphs and constructs a paragraph dataset. The specific expression is as follows: in, represents a paragraph dataset, Indicates the first paragraph divided by the paragraph separator in the text, , No. paragraphs, , No. paragraphs.

[0011] As a preferred technical solution of the present invention, the external data collection module collects all text contents of the current browsing page within 3 days, outputs the subject label of each text, and updates the external data set every 1 day.

[0012] As a preferred technical solution of the present invention, the sentence analysis unit extracts a number of sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs the first similarity factor set in the following specific steps: Step B1: statements The stop words in the sentence are removed, and the words at the end, beginning and middle of the sentence are extracted as the first The comparison words of the sentence; Step B2: The comparison words of each sentence are compared with the external data set one by one. If they are consistent, the first The number of times the comparison words of the sentence are compared is obtained. If they are inconsistent, the comparison of the first sentence is terminated. Comparison of statements; Step B3: Traversal A sentence is formed, and all contrast words are merged and stored to obtain a first similarity factor set. Specifically, all contrast words are merged by accumulating the number of times the same contrast words appear.

[0013] As a preferred technical solution of the present invention, the paragraph analysis unit extracts a number of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs the second similarity factor set in the following specific steps: Step C1: Remove stop words from paragraphs and count the The word that appears most frequently in a paragraph is The comparison words of paragraphs; Step C2: The matching words of each paragraph are compared with the external data set one by one. If they are consistent, the first The number of times the comparison words of the paragraphs are compared is obtained. If they are inconsistent, the comparison of the first paragraph is terminated. Comparison of paragraphs; Step C3: Traversal For each paragraph, all contrasting words are merged and stored to obtain the second similarity factor set.

[0014] As a preferred technical solution of the present invention, the word analysis unit compares the noun data set, the sentiment word data set and the external data set, and outputs the third similarity factor set in a comprehensive manner as follows: Step D1: Compare the first The nouns stored in the noun data are used as the first Noun comparison words are compared with external data sets one by one. If they are consistent, the output is The number of times the noun is compared with the word is obtained. If they are inconsistent, the comparison of the first Comparison of nouns; Step D2: Calculate the The degree of tendency of the noun to contrast with the word , the specific expression is as follows: in, Indicates the distance The sentiment level of the sentiment word with the smallest number of noun characters, Indicates the number of comparisons output in step D1, Indicates the Noun data The frequency of occurrence of the stored nouns; Step D3: traverse nouns and output the tendency of all nouns as the third similarity factor set.

[0015] As a preferred technical solution of the present invention, the text content evaluation module outputs topic keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and the specific expression for text classification is as follows: in, Indicates the Comprehensive value of keywords, 、 、 Respectively represent the first similarity factor set The frequency of keywords, the second similarity factor The frequency of keywords, the third similarity factor The tendency of the keywords is calculated, and the one with the largest comprehensive value is selected as the theme keyword for classification.

[0016] The present invention also provides a text content analysis method combining theme and context, based on the above-mentioned text content analysis system combining theme and context, including the following steps: Step 1: The input text is independently decomposed four times according to sentences, paragraphs and words, and a sentence data set, a paragraph data set, a noun data set and a sentiment word data set are obtained; Step 2: All text content of the current browsing page within 3 days is collected, and the subject label of each text is output to form an external data set; Step 3: A number of sentence keywords are extracted based on the sentence data set, and the sentence keywords are compared with the external data set, and a first similarity factor set is output; Step 4: A number of paragraph keywords are extracted based on the paragraph data set, and the paragraph keywords are compared with the external data set, and a second similarity factor set is output; Step 5: Based on the comparison of the noun data set, the sentiment word data set and the external data set, a third similarity factor set is comprehensively output; Step 6: Based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set and the third similarity factor set, the subject keywords are output and the text is classified.

[0017] Compared with the existing technology, the present invention provides a text content analysis method and system that combines themes and contexts, and has the following beneficial effects: the present invention decomposes the input text four times independently according to sentences, paragraphs and words, and obtains a sentence data set, a paragraph data set, a noun data set and a sentiment word data set, and obtains a first similarity factor set, a second similarity factor set and a third similarity factor set based on the data set, and outputs topic keywords from a comprehensive evaluation of the first similarity factor set, the second similarity factor set and the third similarity factor set, and classifies the text, thereby ensuring the accuracy of keyword extraction in the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the system framework of the present invention; Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] See also Figure 1-2 , a text content analysis system combining theme and context, including a text content disassembly module, an external data acquisition module, a text analysis module and a text content evaluation module; the text content disassembly module includes a sentence disassembly unit, a word disassembly unit and a paragraph disassembly unit, the sentence disassembly unit is used to read the input text, identify the sentence delimiter, and disassemble the text into several sentences according to the sentence delimiter, and construct a sentence data set, the word disassembly unit is used to disassemble the words in each sentence in the sentence data set, and classify the words according to the part of speech to obtain a noun data set and a sentiment word data set, the paragraph disassembly unit is used to read the paragraph delimiter in the text to disassemble the text into several paragraphs, and construct a paragraph data set, the sentence disassembly unit reads the input text, identifies the sentence delimiter, and disassembles the text into several sentences according to the sentence delimiter, and constructs the sentence data set. The specific expression is as follows: in, Represents a statement dataset, Respectively represent the first statement formed by the statement delimiter, , the first statements, , the first Statements, statement separators refer to Table 1 below;

[0021] Table 1

[0022]

[0023] The word decomposition unit decomposes the words in each sentence in the sentence dataset and classifies the words according to the part of speech to obtain the noun dataset and the sentiment word dataset. The specific steps are as follows: Step A1: statements Remove the stop words in statements Several words; Step A2: statements The nouns in several words are classified into the noun dataset, and the sentiment words are classified into the sentiment word dataset until the common The division is completed by the following statement: in, represents a noun dataset, Respectively express the The first noun data obtained by dividing the sentences, , No. Noun data, , No. Noun data, and the Noun data Store nouns and the frequency of noun occurrence. represents the sentiment word dataset, Respectively express the The first sentiment word data obtained by dividing the sentences, , No. Sentiment word data, , No. sentiment word data, and the Sentiment word data The number of characters between the nearest noun and the sentiment level of the sentiment word is stored. The sentiment level of the sentiment word is directly obtained by the natural language processing toolkit NLTK. The paragraph decomposition unit reads the paragraph delimiters in the text to decompose the text into several paragraphs and construct a paragraph dataset. The specific expression is as follows:

[0024] in, represents a paragraph dataset, Indicates the first paragraph divided by the paragraph separator in the text, , No. paragraphs, , No. Paragraphs; see Table 2 for examples of parameters directly obtained through the natural language processing toolkit NLTK;

[0025] Table 2

[0026]

[0027] The external data acquisition module is used to collect all text contents of the current browsing page within 3 days, and output the subject label of each text to form an external data set. The subject label of the text stores the classification mark and text keywords of the text. The external data acquisition module collects all text contents of the current browsing page within 3 days, and outputs the subject label of each text, and updates the external data set every 1 day; the text analysis module includes a sentence analysis unit, a paragraph analysis unit and a word analysis unit. The sentence analysis unit extracts several sentence keywords based on the sentence data set, and compares the sentence keywords with the external data set to output a first similarity factor set. The specific steps of the sentence analysis unit outputting the first similarity factor set are as follows: Step B1: For the first statements The stop words in the sentence are removed, and the words at the end, beginning and middle of the sentence are extracted as the first Step B2: compare the words of the first sentence The comparison words of each sentence are compared with the external data set one by one. If they are consistent, the first The number of times the comparison words of the sentence are compared is obtained. If they are inconsistent, the comparison of the first sentence is terminated. Comparison of statements; Step B3: Traversal Sentences, all contrast words are merged and stored to obtain a first similarity factor set. Specifically, all contrast words are merged by accumulating the number of identical contrast words.

[0028] The paragraph analysis unit extracts a number of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs a second similarity factor set. The specific steps of the paragraph analysis unit outputting the second similarity factor set are as follows: Step C1: Remove stop words from paragraphs and count the The word that appears most frequently in a paragraph is The comparison words of paragraphs; Step C2: The matching words of each paragraph are compared with the external data set one by one. If they are consistent, the first The number of times the comparison words of the paragraphs are compared is obtained. If they are inconsistent, the comparison of the first paragraph is terminated. Comparison of paragraphs; Step C3: Traversal For each paragraph, all contrasting words are merged and stored to obtain the second similarity factor set;

[0029] The word analysis unit compares the noun data set, the sentiment word data set and the external data set, and comprehensively outputs the third similarity factor set. The specific steps of the word analysis unit outputting the third similarity factor set are as follows: Step D1: Compare the first The nouns stored in the noun data are used as the first Noun comparison words are compared with external data sets one by one. If they are consistent, the output is The number of times the noun is compared with the word is obtained. If they are inconsistent, the comparison of the first Comparison of nouns; Step D2: Calculate the The degree of tendency of the noun to contrast with the word , the specific expression is as follows: in, Indicates the distance The sentiment level of the sentiment word with the smallest number of noun characters, Indicates the number of comparisons output in step D1, Indicates the Noun data The frequency of occurrence of the stored nouns; Step D3: traverse nouns and output the tendency of all nouns as the third similarity factor set;

[0030] The text content evaluation module outputs topic keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and classifies the text. The specific expression of the text content evaluation module for text classification is as follows: in, Indicates the Comprehensive value of keywords, 、 、 Respectively represent the first similarity factor set The frequency of keywords, the second similarity factor The frequency of keywords, the third similarity factor The tendency degree of each keyword is calculated, and the keyword with the largest comprehensive value is selected as the theme keyword for classification, thereby ensuring the accuracy of keyword extraction in the text.

[0031] The present invention also provides a text content analysis method combining theme and context, based on the above-mentioned text content analysis system combining theme and context, including the following steps: Step 1: The input text is independently decomposed four times according to sentences, paragraphs and words, and a sentence data set, a paragraph data set, a noun data set and a sentiment word data set are obtained; Step 2: All text content of the current browsing page within 3 days is collected, and the subject label of each text is output to form an external data set; Step 3: A number of sentence keywords are extracted based on the sentence data set, and the sentence keywords are compared with the external data set, and a first similarity factor set is output; Step 4: A number of paragraph keywords are extracted based on the paragraph data set, and the paragraph keywords are compared with the external data set, and a second similarity factor set is output; Step 5: Based on the comparison of the noun data set, the sentiment word data set and the external data set, a third similarity factor set is comprehensively output; Step 6: Based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set and the third similarity factor set, the subject keywords are output and the text is classified.

[0032] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A text content analysis system combining theme and context, characterized by: It includes text content disassembly module, external data collection module, text analysis module and text content evaluation module; The text content disassembly module includes a sentence disassembly unit, a word disassembly unit and a paragraph disassembly unit. The sentence disassembly unit is used to read the input text, identify sentence delimiters, and disassemble the text into several sentences according to the sentence delimiters, and construct a sentence data set. The word disassembly unit is used to disassemble the words in each sentence in the sentence data set, and classify the words according to part of speech to obtain a noun data set and a sentiment word data set. The paragraph disassembly unit is used to read the paragraph delimiters in the text, disassemble the text into several paragraphs, and construct a paragraph data set. The external data collection module is used to collect all text contents of the current browsing page within 3 days and output the subject label of each text to form an external data set. The subject label of the text stores the classification mark and text keywords of the text; The text analysis module includes a sentence analysis unit, a paragraph analysis unit and a word analysis unit. The sentence analysis unit extracts a plurality of sentence keywords based on a sentence data set, compares the sentence keywords with an external data set, and outputs a first similarity factor set. The paragraph analysis unit extracts a plurality of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs a second similarity factor set; The word analysis unit compares the noun dataset, the sentiment word dataset and the external dataset, and comprehensively outputs a third similarity factor set; The specific steps of the word analysis unit comparing the noun dataset, the sentiment word dataset, and the external dataset and comprehensively outputting the third similarity factor set are as follows: Step D1: Compare The nouns stored in the noun data are used as the first Noun comparison words are compared with external data sets one by one. If they are consistent, the output is The number of times the noun is compared with the word is obtained. If they are inconsistent, the comparison of the first Comparison of nouns; Step D2: Calculate the The degree of tendency of the noun to contrast with the word , the specific expression is as follows: in, Indicates the distance The sentiment level of the sentiment word with the smallest number of noun characters, Indicates the number of comparisons output in step D1, Indicates the Noun data The frequency of occurrence of stored nouns; Step D3: Traverse nouns and output the tendency of all nouns as the third similarity factor set; The text content evaluation module outputs topic keywords based on comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and classifies the text; The text content evaluation module outputs topic keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and the specific expression for text classification is as follows: in, Indicates the Comprehensive value of keywords, 、 、 Respectively represent the first similarity factor set The frequency of keywords, the second similarity factor The frequency of keywords, the third similarity factor The tendency of the keywords is calculated, and the one with the largest comprehensive value is selected as the theme keyword for classification.

2. A text content analysis system combining theme and context according to claim 1, characterized in that: The sentence decomposition unit reads the input text, identifies the sentence delimiters, decomposes the text into several sentences based on the sentence delimiters, and constructs the sentence data set. The specific expression is as follows: in, Represents a statement dataset, Respectively represent the first statement formed by the statement delimiter, , the first statements, , the first statements.

3. The text content analysis system combining theme and context according to claim 2, characterized in that: The specific steps of the word decomposition unit decomposing the words in each sentence in the sentence dataset and classifying the words according to parts of speech to obtain a noun dataset and a sentiment word dataset are as follows: Step A1: statements Remove the stop words in statements Several words of Step A2: statements The nouns in several words are classified into the noun dataset, and the sentiment words are classified into the sentiment word dataset until the common The division is completed by the following statement: in, represents a noun dataset, Respectively express the The first noun data obtained by dividing the sentences, , No. Noun data, , No. Noun data, and the Noun data Store nouns and the frequency of noun occurrence. represents the sentiment word dataset, Respectively express the The first sentiment word data obtained by dividing the sentences, , No. Sentiment word data, , No. sentiment word data, and the Sentiment word data The number of characters between the nearest noun and the sentiment level of the sentiment word is stored. The sentiment level of the sentiment word is directly obtained by the natural language processing toolkit NLTK.

4. The text content analysis system combining theme and context according to claim 3, characterized in that: The paragraph decomposition unit reads the paragraph delimiters in the text to decompose the text into several paragraphs and constructs a paragraph dataset. The specific expression is as follows: in, represents a paragraph dataset, Indicates the first paragraph divided by the paragraph separator in the text, , No. paragraphs, , No. paragraphs.

5. The text content analysis system combining theme and context according to claim 4, characterized in that: The external data collection module collects all text contents of the current browsing page within 3 days, outputs the subject label of each text, and updates the external data set every 1 day.

6. The text content analysis system combining theme and context according to claim 5, characterized in that: The sentence analysis unit extracts a number of sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs a first similarity factor set in the following specific steps: Step B1: statements The stop words in the sentence are removed, and the words at the end, beginning and middle of the sentence are extracted as the first The comparison words of each sentence; Step B2: The comparison words of each sentence are compared with the external data set one by one. If they are consistent, the first The number of times the comparison words of the sentence are compared is obtained. If they are inconsistent, the comparison of the first sentence is terminated. Comparison of sentences; Step B3: Traverse A sentence is formed, and all contrast words are merged and stored to obtain a first similarity factor set. Specifically, all contrast words are merged by accumulating the number of times the same contrast words appear.

7. The text content analysis system combining theme and context according to claim 6, characterized in that: The paragraph analysis unit extracts a number of paragraph keywords based on the paragraph dataset, compares the paragraph keywords with the external dataset, and outputs the second similarity factor set in the following specific steps: Step C1: Remove stop words from paragraphs and count the The word that appears most frequently in a paragraph is The comparison words of each paragraph; Step C2: The matching words of each paragraph are compared with the external data set one by one. If they are consistent, the first The number of times the comparison words of the paragraphs are compared is obtained. If they are inconsistent, the comparison of the first paragraph is terminated. Comparison of paragraphs; Step C3: Traverse For each paragraph, all contrasting words are merged and stored to obtain the second similarity factor set.

8. A method for analyzing text content combining theme and context, based on a system for analyzing text content combining theme and context as claimed in any one of claims 1 to 7, characterized in that: The following steps are involved: Step 1: Decompose the input text into sentences, paragraphs, and words four times independently, and obtain sentence datasets, paragraph datasets, noun datasets, and sentiment word datasets; Step 2: Collect all text content of the current browsing page within 3 days and output the topic label of each text to form an external dataset; Step 3: Extract several sentence keywords based on the sentence dataset, compare the sentence keywords with the external dataset, and output a first similarity factor set; Step 4: Extract several paragraph keywords based on the paragraph dataset, compare the paragraph keywords with the external dataset, and output the second similarity factor set; Step 5: Compare the noun dataset, sentiment word dataset, and external dataset, and output the third similarity factor set; Step 6: Based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, output the topic keywords and classify the text.

Images