A method for processing English text data based on data analysis

Abstract

The invention discloses an English text data processing method based on data analysis, which belongs to the technical field of text recommendation, and specifically comprises: obtaining browsed data and unbrowsed data of a target object; determining a target text according to the browsed data, extracting a summary text in the target text and performing a search with a preset signal word, obtaining all text information after the preset signal word, and marking the text information as a conclusion text; performing text analysis on the unbrowsed data to obtain a number of summary texts, performing text duplicate checking calculation on the summary text and the comprehensive conclusion text, and sorting the unbrowsed data from large to small according to the text duplicate checking value, and recommending the unbrowsed data to the target object according to the sorting result. The invention improves the search efficiency and reduces the burden of users screening massive documents.

Description

Technical Field

[0001] The present invention relates to the technical field of text recommendation, and in particular to an English text data processing method based on data analysis. Background Art

[0002] With the development of science and technology, users' demand for English literature and the frequency of queries are increasing year by year. With the advent of the information age, people rely on the Internet to obtain the information they need, but the information on it is growing explosively. How to effectively filter out the required useful information from the massive English information has become a key technical issue.

[0003] Normally, readers pay attention to some key information in the text, such as the title, keywords, abstract, etc., and judge whether it is worthwhile to further read the full text of the document based on this key information. For example, readers click on the corresponding link, open the corresponding document, and then read the abstract to find the key information. If they determine that the document is valuable, they will browse it. However, when a large number of documents need to be viewed, multiple repeated operations will be performed, which is cumbersome and time-consuming, inefficient, and more likely to cause reading fatigue. Summary of the invention

[0004] The purpose of the present invention is to provide an English text data processing method based on data analysis to solve the following technical problems:

[0005] Normally, readers pay attention to some key information in the text, such as the title, keywords, abstract, etc., and judge whether it is worthwhile to further read the full text of the document based on this key information. For example, readers click on the corresponding link, open the corresponding document, and then read the abstract to find the key information. If they determine that the document is valuable, they will browse it. However, when a large number of documents need to be viewed, multiple repeated operations will be performed, which is cumbersome and time-consuming, inefficient, and more likely to cause reading fatigue.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A method for processing English text data based on data analysis comprises the following steps:

[0008] S1, obtaining browsed data and unbrowsed data of the target object after retrieval, wherein the browsed data is the English documents browsed after retrieval, the document browsing time and the document download record; the unbrowsed data is the set of English documents that have not been browsed after retrieval;

[0009] S2, determining the target text according to the browsing data, extracting the summary text in the target text and searching for the preset signal words, obtaining all the preset signal word positions in the summary text, obtaining all the text information after the preset signal words, and marking the text information as the conclusion text; obtaining several conclusion texts in the summary text and integrating the several conclusion texts through natural language processing technology to obtain a comprehensive conclusion text;

[0010] S3, perform text analysis on the unbrowsed data to obtain several summary texts, select any summary text and comprehensive conclusion text in turn to perform text duplication calculation, obtain several text duplication values ​​and mark them as X1, X2, ..., XN, sort the unbrowsed data from large to small according to the text duplication values ​​and recommend unbrowsed data to the target object according to the sorting results.

[0011] As a further solution of the present invention, the S1 further includes:

[0012] The literature database is screened according to the search terms entered by the target object to obtain the search data, which are then sorted according to the publication order of the English literature, the relevance of the article title to the search terms, and the impact factor.

[0013] As a further solution of the present invention, in S2, the process of determining the target text is as follows:

[0014] Obtain the document browsing time t and document download record N of any browsed document. If N is greater than 0, calculate the recommendation score W of all browsed documents according to the calculation formula W=b*t*d, where b and d are preset coefficients and b is less than d; if N is equal to 0, calculate the recommendation score of all browsed documents according to the calculation formula W=b*t; if there is any browsed document whose recommendation score W is greater than or equal to the preset threshold, mark the browsed document as the target text.

[0015] As a further solution of the present invention, in S2, if there is no preset signal word in the summary text, a preset portion of text information is extracted from the summary text, and the text information is used as the conclusion text in the summary text.

[0016] As a further solution of the present invention, in S2, the specific generation process of the comprehensive conclusion text is:

[0017] Preprocessing is performed on several conclusion texts in turn, and the preprocessing includes removing stop words, part-of-speech tagging, and phrase extraction; a semantic relationship graph between the conclusion texts is constructed using semantic network technology, and the several conclusion texts are integrated into a comprehensive conclusion text using NLP technology based on the semantic relationship graph.

[0018] As a further solution of the present invention, the S3 further includes preprocessing a plurality of summary texts, and the specific process is as follows:

[0019] Preset signal word retrieval is performed on any summary text in turn to obtain the positions of all preset signal words in the summary text, obtain the text information after all the preset signal words, and mark the text information as conclusion text, obtain several conclusion texts in the summary text and integrate them through natural language processing technology to obtain a comprehensive conclusion text.

[0020] As a further solution of the present invention, in S3, the specific process of text duplication calculation is as follows:

[0021] The summary text and the comprehensive conclusion text are respectively converted into two groups of ASCII code values ​​in character order; the features of each character are represented by the ASCII code values ​​of the surrounding characters; the features of each character point are reduced in dimension by using an autoencoder; the similarity between the two texts is calculated using the Jaccard similarity coefficient for the reduced-dimensional features;

[0022] The calculation formula for the text duplication value is:

[0023]

[0024] Among them, A is the feature set of comprehensive text, B is the feature set of summary text corresponding to any unbrowsed data, and J is the similarity value.

[0025] As a further solution of the present invention, if there are several target texts, the summary text corresponding to any unbrowsed data is selected and sequentially compared with the comprehensive conclusion texts corresponding to several target texts to perform total text duplication check calculation, the unbrowsed data is sorted from large to small according to the total text duplication check value, and the unbrowsed data is recommended for the target object according to the sorting result;

[0026] The calculation process of the total text duplication value is as follows:

[0027]

[0028] Among them, Ai is the feature set of the comprehensive conclusion text corresponding to any target text, B is the feature set of the summary text corresponding to any unbrowsed data, J' is the total text duplicate check value, Wai is the recommendation score of the comprehensive text corresponding to the target text Ai, and n is the number of target texts.

[0029] Beneficial effects of the present invention:

[0030] The present invention first obtains the target text by analyzing the browsing data of the user to obtain the text browsing direction of the target object, and obtains the summary text of the target text. It can be understood that the text summary is intended to provide the main viewpoints, findings, methods and conclusions of the full text. By retrieving preset signal words and marking the conclusion text, the key conclusion information in the summary text can be quickly extracted, and multiple conclusion texts are integrated using natural language generation technology to obtain a comprehensive conclusion text containing the key conclusion information from the target text. By calculating the similarity between the summary text of unbrowsed data and the comprehensive conclusion text, and sorting them from large to small according to the similarity, the document most relevant to the browsing direction of the target object can be preferentially recommended, which can not only improve the reading interest of the user, but also ensure that the user is exposed to high-quality and relevant document resources, improve the retrieval efficiency, reduce the burden of users screening massive documents, and effectively alleviate the problem of reading fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] Figure 1 It is a flow chart of an English text data processing method based on data analysis of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] See also Figure 1 As shown, the present invention is an English text data processing method based on data analysis, comprising the following steps:

[0035] S1, obtaining browsed data and unbrowsed data of the target object after retrieval, wherein the browsed data is the English documents browsed after retrieval, the document browsing time and the document download record; the unbrowsed data is the set of English documents that have not been browsed after retrieval;

[0036] S2, determining the target text according to the browsing data, extracting the summary text in the target text and searching for the preset signal words, obtaining all the preset signal word positions in the summary text, obtaining all the text information after the preset signal words, and marking the text information as the conclusion text; obtaining several conclusion texts in the summary text and integrating the several conclusion texts through natural language processing technology to obtain a comprehensive conclusion text;

[0037] S3, perform text analysis on the unbrowsed data to obtain several summary texts, select any summary text and comprehensive conclusion text in turn to perform text duplication calculation, obtain several text duplication values ​​and mark them as X1, X2, ..., XN, sort the unbrowsed data from large to small according to the text duplication values ​​and recommend unbrowsed data to the target object according to the sorting results.

[0038] The present invention first obtains the target text by analyzing the browsing data of the user to obtain the text browsing direction of the target object, and obtains the summary text of the target text. It can be understood that the text summary is intended to provide the main viewpoints, findings, methods and conclusions of the full text. By retrieving preset signal words and marking the conclusion text, the key conclusion information in the summary text can be quickly extracted, and multiple conclusion texts are integrated using natural language generation technology to obtain a comprehensive conclusion text containing the key conclusion information from the target text. By calculating the similarity between the summary text of unbrowsed data and the comprehensive conclusion text, and sorting them from large to small according to the similarity, the document most relevant to the browsing direction of the target object can be preferentially recommended, which can not only improve the reading interest of the user, but also ensure that the user is exposed to high-quality and relevant document resources, improve the retrieval efficiency, reduce the burden of users screening massive documents, and effectively alleviate the problem of reading fatigue.

[0039] It is worth noting that the background technology of the present invention indicates that the present invention is based on the need to view a large number of documents, so it is only necessary to obtain the browsing data of the target object, and when the browsing data of the target object reaches a certain threshold, it can be determined whether the target object needs to view a large number of documents;

[0040] Normally, readers pay attention to some key information in the text, such as the title, keywords, abstract, etc., and judge whether it is worthwhile to further read the full text of the document based on these key information. For example, readers click on the corresponding link, open the corresponding document, and then read the abstract to find the key information therein. If the document is valuable, they will browse it. Some users may tend to browse for a long time without downloading, while some users may tend to download without in-depth browsing. Therefore, the present invention determines the target text according to the browsing time and the number of downloads;

[0041] It is understandable that the text summary is intended to provide the main ideas, findings, methods and conclusions of the full text, which is a summary of the literature. Therefore, the summary text information of the target text is extracted and the conclusion is extracted. It is understandable that for the purpose of introduction, some existing literature technologies are usually summarized in the summary, which has little actual reference value. Therefore, conclusion extraction is performed and conclusion extraction is performed on the browsing data, and text duplication is checked based on the conclusion extraction. It is understandable that if the summary text conclusion is not extracted and the text duplication check is performed directly, not only will the amount of calculation be large, but there will also be a large number of literature technologies written for the purpose of introduction in the summary text for summary. When there are similar literature technology summaries, it will affect the final text duplication check results, resulting in inaccurate final recommendation results.

[0042] In a preferred embodiment of the present invention, the S1 further includes:

[0043] The literature database is screened according to the search terms entered by the target object to obtain the search data, which are then sorted according to the publication order of the English literature, the relevance of the article title to the search terms, and the impact factor.

[0044] In another preferred embodiment of the present invention, in S2, the process of determining the target text is:

[0045] Obtain the document browsing time t and document download record N of any browsed document. If N is greater than 0, calculate the recommendation score W of all browsed documents according to the calculation formula W=b*t*d, where b and d are preset coefficients and b is less than d; if N is equal to 0, calculate the recommendation score of all browsed documents according to the calculation formula W=b*t; if there is any browsed document whose recommendation score W is greater than or equal to the preset threshold, mark the browsed document as the target text.

[0046] Normally, readers pay attention to some key information in the text, such as the title, keywords, abstract, etc., and judge whether it is worthwhile to further read the full text of the document based on this key information. For example, readers click on the corresponding link, open the corresponding document, and then read the abstract to find the key information. If they determine that the document is valuable, they will browse it. Some users may tend to browse for a long time without downloading, while others may tend to download without in-depth browsing. Whether the document meets the needs of the target object is measured by browsing time and number of downloads.

[0047] In another preferred embodiment of the present invention, in S2, if there is no preset signal word in the summary text, a preset portion of text information is extracted from the summary text, and the text information is used as the conclusion text in the summary text.

[0048] It is understandable that in many abstracts of English documents, there are no clear signal words or keyword prompts in the conclusion part, or in some abstracts of English documents, there is no conclusion part at all, and it must be judged by semantic understanding. When summarizing the content of the abstract of such English documents, since the content of the abstract summary is generally placed at the end of the entire abstract, it can be considered to obtain it from the latter part of the English abstract information, specifically, from the back to the front in the English abstract information, obtain a preset number of sentences, and use the preset number of sentences (for example, it can be 3 sentences, that is, the sentence information of the first to last sentence, the second to last sentence and the third to last sentence) as the conclusion information in the English abstract information.

[0049] In another preferred embodiment of the present invention, in S2, the specific generation process of the comprehensive conclusion text is:

[0050] Preprocessing is performed on several conclusion texts in turn, and the preprocessing includes removing stop words, part-of-speech tagging, and phrase extraction; a semantic relationship graph between the conclusion texts is constructed using semantic network technology, and the several conclusion texts are integrated into a comprehensive conclusion text using NLP technology based on the semantic relationship graph.

[0051] In another preferred embodiment of the present invention, the step S3 further includes preprocessing a plurality of summary texts, and the specific process is as follows:

[0052] Preset signal word retrieval is performed on any summary text in turn to obtain the positions of all preset signal words in the summary text, any preset signal word is selected to obtain the text information after the preset signal word, and the text information is marked as the conclusion text, and several conclusion texts in the summary text are obtained and integrated through natural language generation technology to obtain a comparative comprehensive conclusion text.

[0053] It is understandable that a standard paper abstract usually contains the purpose, method, conclusion and result of the author of the document, and the purpose is usually derived based on the deficiencies of the existing technology. When the purpose in any unprocessed abstract text contains the conclusion information in the target text, the abstract text is similar to the comprehensive conclusion text in the duplicate check, but it is not actually required by the target object, resulting in the final recommendation effect is not obvious; secondly, if the abstract text is not preprocessed, it will lead to the need to transcode each character in the abstract text, which contains a large amount of unnecessary text, which not only increases the data processing task volume, but also increases the computational complexity of subsequent text duplicate checking.

[0054] The preset signal word search can efficiently locate the key information in the abstract text, avoiding the tedious process of manual screening and extraction. The natural language generation technology can integrate multiple conclusion texts to form a coherent comprehensive conclusion, which improves the readability and comprehension of the text.

[0055] In another preferred embodiment of the present invention, in S3, the specific process of text duplication checking calculation is:

[0056] The summary text and the comprehensive conclusion text are respectively converted into two groups of ASCII code values ​​in character order; the features of each character are represented by the ASCII code values ​​of the surrounding characters; the features of each character point are reduced in dimension by using an autoencoder; the similarity between the two texts is calculated using the Jaccard similarity coefficient for the reduced-dimensional features;

[0057] The calculation formula for the text duplication value is:

[0058]

[0059] Among them, A is the feature set of the comprehensive text, B is the feature set of any summary text, and J is the similarity value.

[0060] In another preferred embodiment of the present invention, if there are several target texts, the summary text corresponding to any unbrowsed data is selected and sequentially compared with the comprehensive conclusion texts corresponding to several target texts to perform total text duplication check calculation, the unbrowsed data is sorted from large to small according to the total text duplication check value, and the unbrowsed data is recommended for the target object according to the sorting result;

[0061] The calculation process of the total text duplication value is as follows:

[0062]

[0063] Among them, Ai is the feature set of the comprehensive conclusion text corresponding to any target text, B is the feature set of the summary text corresponding to any unbrowsed data, J' is the total text duplicate check value, Wai is the recommendation score of the comprehensive text corresponding to the target text Ai, and n is the number of target texts.

[0064] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for processing English text data based on data analysis, characterized in that: The following steps are involved: S1, obtaining browsed data and unbrowsed data of the target object after retrieval, wherein the browsed data is the English documents browsed after retrieval, the document browsing time and the document download record; the unbrowsed data is the set of English documents that have not been browsed after retrieval; S2, determining the target text according to the browsing data, extracting the summary text in the target text and searching for the preset signal words, obtaining all the preset signal word positions in the summary text, obtaining all the text information after the preset signal words, and marking the text information as conclusion text to obtain a plurality of conclusion texts; obtaining a plurality of conclusion texts in the summary text and integrating the plurality of conclusion texts through natural language processing technology to obtain a comprehensive conclusion text; S3, performing text analysis on the unbrowsed data to obtain a number of summary texts, selecting any summary text and comprehensive conclusion text in turn to perform text duplication calculation, obtaining a number of text duplication values ​​and marking them as X1, X2, ..., XN, sorting the unbrowsed data from large to small according to the text duplication values, and recommending unbrowsed data for the target object according to the sorting results; The specific process of text duplication calculation is as follows: The summary text and the comprehensive conclusion text are respectively converted into two groups of ASCII code values ​​in character order; the characteristics of each character are represented by the ASCII code values ​​of the surrounding characters; The autoencoder is used to reduce the dimension of each character point; the Jaccard similarity coefficient is used to calculate the similarity between two texts after the dimension reduction. The calculation formula for the text duplication value is: ; Among them, A is the feature set of comprehensive text, B is the feature set of summary text corresponding to any unbrowsed data, and J is the similarity value; If there are several target texts, the summary text corresponding to any unbrowsed data is selected and compared with the comprehensive conclusion texts corresponding to several target texts in turn to calculate the total text duplication. The unbrowsed data are sorted from large to small according to the total text duplication value, and the unbrowsed data are recommended for the target object according to the sorting result. The calculation process of the total text duplication value is: ; Among them, Ai is the feature set of the comprehensive conclusion text corresponding to any target text, B is the feature set of the summary text corresponding to any unbrowsed data, J' is the total text duplicate check value, Wai is the recommendation score of the comprehensive text corresponding to the target text Ai, and n is the number of target texts.

2. The English text data processing method based on data analysis according to claim 1 is characterized in that: Said S1 further includes: The literature database is screened according to the search terms entered by the target object to obtain the search data, which are then sorted according to the publication order of the English literature, the relevance of the article title to the search terms, and the impact factor.

3. The English text data processing method based on data analysis according to claim 1 is characterized in that: In S2, the process of determining the target text is as follows: Obtain the document browsing time t and document download record N of any browsed document. If N is greater than 0, calculate the recommendation score W of all browsed documents according to the calculation formula W=b*t*d, where b and d are preset coefficients and b is less than d; If N is equal to 0, the recommendation scores of all browsed documents are calculated according to the calculation formula W=b*t; If there is any browsed document whose recommendation score W is greater than or equal to a preset threshold, the browsed document is marked as the target text.

4. The English text data processing method based on data analysis according to claim 1 is characterized in that: In S2, if the preset signal word does not exist in the summary text, a preset portion of text information is extracted from the summary text, and the text information is used as the conclusion text in the summary text.

5. The English text data processing method based on data analysis according to claim 1 is characterized in that: In S2, the specific generation process of the comprehensive conclusion text is: Preprocessing is performed on several conclusion texts in turn, and the preprocessing includes removing stop words, part-of-speech tagging, and phrase extraction; a semantic relationship graph between the conclusion texts is constructed using semantic network technology, and the several conclusion texts are integrated into a comprehensive conclusion text using NLP technology based on the semantic relationship graph.

6. The English text data processing method based on data analysis according to claim 1 is characterized in that: The S3 also includes preprocessing of a number of summary texts, and the specific process is as follows: Preset signal word retrieval is performed on any summary text in turn to obtain the positions of all preset signal words in the summary text, obtain the text information after all the preset signal words, and mark the text information as conclusion text, obtain several conclusion texts in the summary text and integrate them through natural language processing technology to obtain a comprehensive conclusion text.

Images