Text content analysis method and system combining theme and context

Abstract

The invention relates to the technical field of text data analysis, and discloses a text content analysis method and system combining themes and contexts. The system comprises a text content disassembly module, an external data acquisition module, a text analysis module and a text content evaluation module. The text content splitting module comprises a statement splitting unit, a word splitting unit and a paragraph splitting unit, and the text analysis module comprises a statement analysis unit, a paragraph analysis unit and a word analysis unit. According to the system, an input text is independently disassembled for four times according to statements, paragraphs and words, a statement data set, a paragraph data set, a noun data set and an emotion word data set are obtained, and a first similar factor set, a second similar factor set and a third similar factor set are obtained based on the data sets; and the subject keywords are comprehensively evaluated and output from the first similarity factor set, the second similarity factor set and the third similarity factor set, and the texts are classified, so that the accuracy of keyword extraction in the texts is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of text data analysis, and in particular to a text content analysis method and system combining subject and context. Background Art

[0002] With the rapid development of the Internet and the widespread popularity of personal computers, more and more messages and data are published in the form of electronic documents through the hypertext transfer protocol. As a result, the speed and ability of data retrieval have encountered huge challenges. How to accurately and timely obtain the information required by users in the vast world of information has become a major problem in the development of the Internet.

[0003] Search engine technology is a relatively mature information retrieval technology. However, with the crazy growth of Internet data, the shortcomings of search engines have gradually emerged. Traditional search engines can usually only use keywords as input. For ordinary users, it is often difficult for them to condense a small number of keywords to accurately express their query intentions. In addition, the search engine's return result is not a concise and accurate answer, but a list of web page fragments. These web page fragments usually contain a lot of noise data. Users still need to read these web page fragments and even the corresponding original web pages to find the answers they need. However, when users upload, it is easy for the keywords in the text subject to deviate from the actual keywords in the text, resulting in errors in keyword classification, which leads to low efficiency in text retrieval by users. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies in the prior art, the present invention provides a text content analysis method and system that combines theme and context, which has the advantages of accurately extracting keywords from text and solves the above-mentioned technical problems.

[0006] (II) Technical solution

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a text content analysis system combining theme and context, comprising a text content disassembly module, an external data collection module, a text analysis module and a text content evaluation module; the text content disassembly module comprises a sentence disassembly unit, a word disassembly unit and a paragraph disassembly unit, the sentence disassembly unit is used to read the input text, and identify the sentence delimiter, and according to the sentence delimiter, the text is disassembled into a plurality of sentences, and a sentence data set is constructed, the word disassembly unit is used to disassemble the words in each sentence in the sentence data set, and classify the words according to the part of speech to obtain a noun data set and a sentiment word data set, the paragraph disassembly unit is used to read the paragraph delimiter in the text to disassemble the text into a plurality of paragraphs, and a paragraph data set is constructed; the external data collection module is used to collect all text within 3 days of the current browsing page content, and outputs the subject label of each text to constitute an external data set, wherein the subject label of the text stores the classification mark and text keywords of the text; the text analysis module includes a sentence analysis unit, a paragraph analysis unit and a word analysis unit, wherein the sentence analysis unit extracts a number of sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs a first similarity factor set; the paragraph analysis unit extracts a number of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs a second similarity factor set; the word analysis unit compares the noun data set, the sentiment word data set and the external data set, and comprehensively outputs a third similarity factor set; the text content evaluation module comprehensively evaluates and outputs the subject keywords based on the first similarity factor set, the second similarity factor set and the third similarity factor set, and classifies the text.

[0008] As a preferred technical solution of the present invention, the sentence decomposition unit reads the input text, identifies the sentence separator, decomposes the text into several sentences according to the sentence separator, and constructs the specific expression of the sentence data set as follows: in, Represents a statement dataset, Respectively represent the first statement formed by the statement separator, , the first Sentences, , the first statements.

[0009] As a preferred technical solution of the present invention, the word decomposition unit decomposes the words in each sentence in the sentence data set, and classifies the words according to the part of speech to obtain the noun data set and the sentiment word data set. The specific steps are as follows: Step A1: Statements Remove the stop words in Statements Step A2: Statements The nouns in several words are classified into the noun dataset, and the sentiment words are classified into the sentiment word dataset until the common The division is completed by the statement, and its specific expression is as follows: in, represents a noun dataset, Respectively express the The first noun data obtained by dividing the sentences, , No. Noun data, , No. Noun data, and the Noun data Stores nouns and the frequency of their occurrence. represents the sentiment word dataset, Respectively express the The first sentiment word data obtained by dividing sentences, , No. Sentiment word data, , No. sentiment word data, and the Sentiment word data The number of characters of the nearest noun distance of the sentiment word and the sentiment level of the sentiment word are stored, and the sentiment level of the sentiment word is directly obtained by the natural language processing toolkit NLTK.

[0010] As a preferred technical solution of the present invention, the paragraph decomposition unit reads the paragraph separators in the text to decompose the text into several paragraphs, and constructs a paragraph data set. The specific expression is as follows: in, represents a paragraph dataset, Indicates the first paragraph divided by the paragraph separator in the text. , No. paragraphs, , No. paragraphs.

[0011] As a preferred technical solution of the present invention, the external data collection module collects all text contents of the current browsing page within 3 days, outputs the subject label of each text, and updates the external data set every 1 day.

[0012] As a preferred technical solution of the present invention, the sentence analysis unit extracts a number of sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs the first similarity factor set in the following specific steps: Step B1: Statements The stop words in the sentence are removed, and the words at the end, beginning and middle of the sentence are extracted as the first Step B2: compare the words of the first sentence The comparison words of each sentence are compared with the external data set one by one. If they are consistent, the output is The number of times the comparison words of the sentences are compared is obtained. If they are inconsistent, the comparison of the first Comparison of statements; Step B3: Traversal A sentence is formed, and all the contrast words are merged and stored to obtain a first similarity factor set. Specifically, all the contrast words are merged in that the number of times the same contrast words appear is accumulated.

[0013] As a preferred technical solution of the present invention, the paragraph analysis unit extracts a number of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs the second similarity factor set in the following specific steps: Step C1: Remove stop words from paragraphs and count the The most frequent word in the paragraph is taken as the The comparison words of paragraphs; Step C2: The matching words of each paragraph are compared with the external data set one by one. If they are consistent, the output is The number of times the matching words of the paragraphs are compared is obtained. If they are inconsistent, the comparison of the first paragraph is terminated. Comparison of paragraphs; Step C3: Traversal paragraphs, merge and store all contrasting words to obtain the second similarity factor set.

[0014] As a preferred technical solution of the present invention, the specific steps of the word analysis unit comparing the noun data set, the sentiment word data set and the external data set to comprehensively output the third similarity factor set are as follows: Step D1: Compare the first The nouns stored in the noun data are used as the first Noun comparison words are compared with external data sets one by one. If they are consistent, the output is The number of times the noun is compared with the word, if they are inconsistent, the comparison of the first comparison of nouns; Step D2: Calculate the The degree of preference of a noun to a word , the specific expression is as follows: in, Indicates the distance The sentiment level of the sentiment word with the smallest number of noun characters. represents the number of comparisons output in step D1, Indicates Noun data The frequency of occurrence of the stored nouns; Step D3: traverse nouns and output the inclinations of all nouns as the third similarity factor set.

[0015] As a preferred technical solution of the present invention, the text content evaluation module outputs topic keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and the specific expression for text classification is as follows: in, Indicates The comprehensive value of keywords, , , Respectively represent the first similarity factor set The frequency of the keywords, the second similarity factor set The frequency of keywords, the third similarity factor set The tendency of the keywords is calculated, and the one with the largest comprehensive value is selected as the theme keyword for classification.

[0016] The present invention also provides a text content analysis method combining theme and context, based on the above-mentioned text content analysis system combining theme and context, comprising the following steps: step one: decomposing the input text four times independently according to sentences, paragraphs and words, and obtaining a sentence data set, a paragraph data set, a noun data set and a sentiment word data set; step two: collecting all text content of the current browsing page within 3 days, and outputting the subject label of each text to form an external data set; step three: extracting a number of sentence keywords based on the sentence data set, and comparing the sentence keywords with the external data set, and outputting a first similarity factor set; step four: extracting a number of paragraph keywords based on the paragraph data set, and comparing the paragraph keywords with the external data set, and outputting a second similarity factor set; step five: comparing the noun data set, the sentiment word data set and the external data set, and comprehensively outputting a third similarity factor set; step six: comprehensively evaluating and outputting the subject keywords based on the first similarity factor set, the second similarity factor set and the third similarity factor set, and classifying the text.

[0017] Compared with the prior art, the present invention provides a text content analysis method and system that combines theme and context, and has the following beneficial effects: the present invention decomposes the input text four times independently according to sentences, paragraphs and words, and obtains a sentence data set, a paragraph data set, a noun data set and a sentiment word data set, and obtains a first similarity factor set, a second similarity factor set and a third similarity factor set based on the data set, and outputs theme keywords from the first similarity factor set, the second similarity factor set and the third similarity factor set through comprehensive evaluation, and classifies the text, thereby ensuring the accuracy of keyword extraction in the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the system framework of the present invention; Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1-2 , a text content analysis system combining theme and context, including a text content disassembly module, an external data collection module, a text analysis module and a text content evaluation module; the text content disassembly module includes a sentence disassembly unit, a word disassembly unit and a paragraph disassembly unit, the sentence disassembly unit is used to read the input text, identify the sentence delimiter, and disassemble the text into several sentences according to the sentence delimiter, and construct a sentence data set, the word disassembly unit is used to disassemble the words in each sentence in the sentence data set, and classify the words according to the part of speech to obtain a noun data set and a sentiment word data set, the paragraph disassembly unit is used to read the paragraph delimiter in the text to disassemble the text into several paragraphs, and construct a paragraph data set, the sentence disassembly unit reads the input text, identifies the sentence delimiter, and disassembles the text into several sentences according to the sentence delimiter, and constructs the sentence data set. The specific expression is as follows: in, Represents a statement dataset, Respectively represent the first statement formed by the statement separator, , the first Sentences, , the first Statements, statement separators refer to Table 1 below; Table 1

[0021] The word decomposition unit decomposes the words in each sentence in the sentence data set, and classifies the words according to the part of speech to obtain the noun data set and the sentiment word data set. The specific steps are as follows: Step A1: Statements Remove the stop words in Statements Step A2: Statements The nouns in several words are classified into the noun dataset, and the sentiment words are classified into the sentiment word dataset until the common The division is completed by the statement, and its specific expression is as follows: in, represents a noun dataset, Respectively express the The first noun data obtained by dividing the sentences, , No. Noun data, , No. Noun data, and the Noun data Stores nouns and the frequency of their occurrence. represents the sentiment word dataset, Respectively express the The first sentiment word data obtained by dividing sentences, , No. Sentiment word data, , No. sentiment word data, and the Sentiment word data The number of characters between the nearest nouns and the sentiment level of sentiment words are stored. The sentiment level of sentiment words is directly obtained by the natural language processing toolkit NLTK. The paragraph decomposition unit reads the paragraph separators in the text to decompose the text into several paragraphs and construct a paragraph dataset. The specific expression is as follows: in, represents a paragraph dataset, Indicates the first paragraph divided by the paragraph separator in the text. , No. paragraphs, , No. paragraphs; see Table 2 for examples of parameters directly obtained through the natural language processing toolkit NLTK; Table 2

[0022] The external data collection module is used to collect all text contents of the current browsing page within 3 days, and output the subject label of each text to form an external data set. The subject label of the text stores the classification mark and text keywords of the text. The external data collection module collects all text contents of the current browsing page within 3 days, and outputs the subject label of each text, and updates the external data set every 1 day; the text analysis module includes a sentence analysis unit, a paragraph analysis unit and a word analysis unit. The sentence analysis unit extracts a number of sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs a first similarity factor set. The specific steps of the sentence analysis unit outputting the first similarity factor set are as follows: Step B1: For the first Statements The stop words in the sentence are removed, and the words at the end, beginning and middle of the sentence are extracted as the first Step B2: compare the words of the first sentence The comparison words of each sentence are compared with the external data set one by one. If they are consistent, the output is The number of times the comparison words of the sentences are compared is obtained. If they are inconsistent, the comparison of the first Comparison of statements; Step B3: Traversal Sentences, all contrast words are merged and stored to obtain a first similarity factor set. Specifically, all contrast words are merged by accumulating the number of identical contrast words. The paragraph analysis unit extracts a number of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs a second similarity factor set. The specific steps of the paragraph analysis unit outputting the second similarity factor set are as follows: Step C1: Remove stop words from paragraphs and count the The most frequent word in the paragraph is taken as the The comparison words of paragraphs; Step C2: The matching words of each paragraph are compared with the external data set one by one. If they are consistent, the output is The number of times the matching words of the paragraphs are compared is obtained. If they are inconsistent, the comparison of the first paragraph is terminated. Comparison of paragraphs; Step C3: Traversal paragraphs, merge and store all contrasting words to obtain the second similarity factor set; The word analysis unit compares the noun data set, the sentiment word data set and the external data set, and comprehensively outputs the third similarity factor set. The specific steps of the word analysis unit outputting the third similarity factor set are as follows: Step D1: Compare the first The nouns stored in the noun data are used as the first Noun comparison words are compared with external data sets one by one. If they are consistent, the output is The number of times the noun is compared with the word, if they are inconsistent, the comparison of the first comparison of nouns; Step D2: Calculate the The degree of preference of a noun to a word , the specific expression is as follows: in, Indicates the distance The sentiment level of the sentiment word with the smallest number of noun characters. represents the number of comparisons output in step D1, Indicates Noun data The frequency of occurrence of the stored nouns; Step D3: traverse nouns, and output the inclination of all nouns as the third similarity factor set; The text content evaluation module outputs topic keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and classifies the text. The specific expression of the text content evaluation module for text classification is as follows: in, Indicates The comprehensive value of keywords, , , Respectively represent the first similarity factor set The frequency of the keywords, the second similarity factor set The frequency of keywords, the third similarity factor set The tendency of each keyword is calculated, and the keyword with the largest comprehensive value is selected as the theme keyword for classification, thereby ensuring the accuracy of keyword extraction in the text.

[0023] The present invention also provides a text content analysis method combining theme and context, based on the above-mentioned text content analysis system combining theme and context, comprising the following steps: step one: decomposing the input text four times independently according to sentences, paragraphs and words, and obtaining a sentence data set, a paragraph data set, a noun data set and a sentiment word data set; step two: collecting all text content of the current browsing page within 3 days, and outputting the subject label of each text to form an external data set; step three: extracting a number of sentence keywords based on the sentence data set, and comparing the sentence keywords with the external data set, and outputting a first similarity factor set; step four: extracting a number of paragraph keywords based on the paragraph data set, and comparing the paragraph keywords with the external data set, and outputting a second similarity factor set; step five: comparing the noun data set, the sentiment word data set and the external data set, and comprehensively outputting a third similarity factor set; step six: comprehensively evaluating and outputting the subject keywords based on the first similarity factor set, the second similarity factor set and the third similarity factor set, and classifying the text.

[0024] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text content analysis system combining theme and context, characterized by: It includes a text content disassembly module, an external data collection module, a text analysis module and a text content evaluation module; the text content disassembly module includes a sentence disassembly unit, a word disassembly unit and a paragraph disassembly unit, the sentence disassembly unit is used to read the input text, identify the sentence delimiter, and disassemble the text into several sentences according to the sentence delimiter, and construct a sentence data set, the word disassembly unit is used to disassemble the words in each sentence in the sentence data set, and classify the words according to the part of speech to obtain a noun data set and a sentiment word data set, the paragraph disassembly unit is used to read the paragraph delimiter in the text to disassemble the text into several paragraphs, and construct a paragraph data set; the external data collection module is used to collect all text contents of the current browsing page within 3 days, and output the subject label of each text to form an external data set, and the subject label of the text stores the classification mark and text keywords of the text; the text analysis module includes a sentence analysis unit, a paragraph analysis unit and a word analysis unit, the sentence analysis unit extracts several sentence keywords based on the sentence data set, compares the sentence keywords with the external data set, and outputs a first similarity factor set; The paragraph analysis unit extracts a plurality of paragraph keywords based on the paragraph data set, compares the paragraph keywords with the external data set, and outputs a second similarity factor set; The word analysis unit compares the noun data set, the sentiment word data set and the external data set, and comprehensively outputs a third similarity factor set; The text content evaluation module outputs topic keywords based on comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and classifies the text.

2. A text content analysis system combining theme and context according to claim 1, characterized in that: The sentence decomposition unit reads the input text, identifies sentence delimiters, decomposes the text into several sentences according to the sentence delimiters, and constructs the sentence data set. The specific expression is as follows: in, Represents a statement dataset, Respectively represent the first statement formed by the statement separator, , the first Sentences, , the first statements.

3. A text content analysis system combining theme and context according to claim 2, characterized in that: The specific steps of the word decomposition unit decomposing the words in each sentence in the sentence data set and classifying the words according to the part of speech to obtain the noun data set and the sentiment word data set are as follows: Step A1: Statements Remove the stop words in Statements Step A2: Statements The nouns in several words are classified into the noun dataset, and the sentiment words are classified into the sentiment word dataset until the common The division is completed by the statement, and its specific expression is as follows: in, represents a noun dataset, Respectively express the The first noun data obtained by dividing the sentences, , No. Noun data, , No. Noun data, and the Noun data Stores nouns and the frequency of their occurrence. represents the sentiment word dataset, Respectively express the The first sentiment word data obtained by dividing sentences, , No. Sentiment word data, , No. sentiment word data, and the Sentiment word data The number of characters of the nearest noun distance of the sentiment word and the sentiment level of the sentiment word are stored, and the sentiment level of the sentiment word is directly obtained by the natural language processing toolkit NLTK.

4. A text content analysis system combining theme and context according to claim 3, characterized in that: The paragraph decomposition unit reads the paragraph separators in the text to decompose the text into several paragraphs and constructs a paragraph data set. The specific expression is as follows: in, represents a paragraph dataset, Indicates the first paragraph divided by the paragraph separator in the text. , No. paragraphs, , No. paragraphs.

5. A text content analysis system combining theme and context according to claim 4, characterized in that: The external data collection module collects all text contents of the current browsing page within 3 days, outputs the subject label of each text, and updates the external data set every 1 day.

6. A text content analysis system combining theme and context according to claim 5, characterized in that: The specific steps of extracting a plurality of sentence keywords based on the sentence data set, comparing the sentence keywords with the external data set, and outputting the first similarity factor set are as follows: Step B1: Statements The stop words in the sentence are removed, and the words at the end, beginning and middle of the sentence are extracted as the first Step B2: compare the words of the first sentence The comparison words of each sentence are compared with the external data set one by one. If they are consistent, the output is The number of times the comparison words of the sentences are compared is obtained. If they are inconsistent, the comparison of the first Comparison of statements; Step B3: Traversal A sentence is formed, and all the contrast words are merged and stored to obtain a first similarity factor set. Specifically, all the contrast words are merged in that the number of identical contrast words is accumulated.

7. A text content analysis system combining theme and context according to claim 6, characterized in that: The specific steps of extracting a plurality of paragraph keywords based on the paragraph data set and comparing the paragraph keywords with the external data set to output the second similarity factor set are as follows: Step C1: Remove stop words from paragraphs and count the The most frequent word in the paragraph is taken as the The matching words of the paragraph; Step C2: The matching words of each paragraph are compared with the external data set one by one. If they are consistent, the output is The number of times the matching words of the paragraphs are compared is obtained. If they are inconsistent, the comparison of the first paragraph is terminated. Comparison of paragraphs; Step C3: Traversal paragraphs, merge and store all contrasting words to obtain the second similarity factor set.

8. A text content analysis system combining theme and context according to claim 7, characterized in that: The specific steps of the word analysis unit comparing the noun data set, the sentiment word data set and the external data set to comprehensively output the third similarity factor set are as follows: Step D1: Compare the The nouns stored in the noun data are Noun comparison words are compared with external data sets one by one. If they are consistent, the output is The number of times the noun is compared with the word, if they are inconsistent, the comparison of the first comparison of nouns; Step D2: Calculate the The degree of preference of a noun to a word , the specific expression is as follows: in, Indicates the distance The sentiment level of the sentiment word with the smallest number of noun characters. represents the number of comparisons output in step D1, Indicates Noun data The frequency of occurrence of the stored nouns; Step D3: traverse nouns and output the inclinations of all nouns as the third similarity factor set.

9. A text content analysis system combining theme and context according to claim 8, characterized in that: The text content evaluation module outputs topic keywords based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set, and the third similarity factor set, and the specific expression for text classification is as follows: in, Indicates The comprehensive value of keywords, , , Respectively represent the first similarity factor set The frequency of the keywords, the second similarity factor set The frequency of keywords, the third similarity factor set The tendency of the keywords is calculated, and the one with the largest comprehensive value is selected as the theme keyword for classification.

10. A text content analysis method combining theme and context, based on a text content analysis system combining theme and context as claimed in any one of claims 1 to 9, characterized in that: The method comprises the following steps: Step 1: the input text is decomposed four times independently according to sentences, paragraphs and words, and a sentence data set, a paragraph data set, a noun data set and a sentiment word data set are obtained; Step 2: all text contents of the current browsing page within 3 days are collected, and the subject label of each text is output to form an external data set; Step 3: a number of sentence keywords are extracted based on the sentence data set, and the sentence keywords are compared with the external data set, and a first similarity factor set is output; Step 4: a number of paragraph keywords are extracted based on the paragraph data set, and the paragraph keywords are compared with the external data set, and a second similarity factor set is output; Step 5: based on the comparison of the noun data set, the sentiment word data set and the external data set, a third similarity factor set is comprehensively output; Step 6: based on the comprehensive evaluation of the first similarity factor set, the second similarity factor set and the third similarity factor set, the subject keywords are output and the text is classified.

Images