A web page processing method based on key long sentence and text length pre-classification

Abstract

The application relates to the technical field of Internet and discloses a webpage processing method based on key long sentences and text length pre-classification, which comprises the following steps: text information extraction, first, the entire webpage data is processed to extract an HTML tree structure, so that the entire text content in the webpage HTML tree is extracted out. The webpage processing method based on key long sentences and text length pre-classification combines the advantages of key long sentence-based webpage feature code extraction, key long sentence-based webpage feature code extraction and semantic feature extraction technology, simultaneously plays the importance of semantic key long sentences and the connection between contexts, reduces the number of required features, greatly improves the accuracy of feature extraction and the efficiency of duplicate checking, and through the webpage text information extraction method based on Chinese punctuation symbol HTML tree structure, the method has strong universality and high accuracy, and the efficiency of webpage semantic cohesion text processing is improved.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, specifically to a webpage processing method based on pre-classification of key long sentences and text length. Background Technology

[0002] The continued rapid development of the internet has led to an explosive growth in the number of websites and their associated web pages. To enable users to quickly find content of interest amidst this vast amount of information, search engines emerged, their primary mission being to provide users with accurate and efficient search results. However, along with the dramatic increase in the number of web pages, the phenomenon of reposting has also become quite common. Statistics show that the duplication rate of web pages on the internet is as high as 40%.

[0003] Search results often contain many duplicate records. This duplicate information not only increases the storage burden and query efficiency of search engines but also significantly reduces the user experience. Therefore, how to quickly and accurately discover web pages with similar content has become one of the key issues in improving the quality of search engine services, and web page deduplication technology has undoubtedly become the most important part of web page information processing. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a webpage processing method based on pre-classification of key long sentences and text length. This method has advantages such as avoiding duplicate display of webpages and solves the problem that duplicate information not only increases the storage burden and query efficiency of search engines but also greatly reduces the user experience.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A webpage processing method based on pre-classification of key long sentences and body text length includes the following steps:

[0007] S1. Extraction of main text information

[0008] First, the entire webpage data needs to be processed to extract the HTML tree structure. The webpage data source is selected from 10 different portal websites, such as Sina, NetEase, and Sohu. Ten text-based webpages are randomly selected from each website, so that the entire content of the text is extracted from the HTML tree. The text includes the topic and content of the text. The HTML tree is traversed to extract all <text> nodes, and the link from the root node to each <text> leaf node is recorded. Here, a recursive algorithm is used to traverse the HTML tree.

[0009] S2. Determination of Text Structure Samples

[0010] The system checks the number of Chinese periods in each text segment and selects the link corresponding to the text segment with the largest number of periods as the structural sample. The text segment with the largest number of periods refers to the text segment containing the most periods in the selected text. Other text segments with a large number of Chinese periods are selected and their link structures are compared with the structural sample to determine the correctness of the structural sample.

[0011] S3, Webpage Feature Selection Threshold

[0012] For the main text content, the similarity threshold is chosen primarily considering frequently occurring links and modified text blocks within the content. Taking into account the worst-case scenario where some text blocks are simultaneously linked and modified, the number of nodes whose link structure matches the sample link structure is reduced by 1, and the link length is increased by 2, resulting in the following similarity:

[0013]

[0014] For example: when the link length of the L0 sample text is 11,

[0015] Its K0 is calculated using the above formula, K0 = 0.83, so the similarity between link i and link j is 0.83;

[0016] When the link length of the L0 sample text is 18,

[0017] Its K0 is calculated using the above formula, K0 = 0.89, so the similarity between link i and link j is 0.89;

[0018] When the link length of the L0 sample text is 4,

[0019] K0 is calculated using the above formula, and K0 = 0.6, so the similarity between link i and link j is 0.6.

[0020] S4. Webpage Feature Extraction Based on Key Long Sentences

[0021] A key long sentence refers to the longest sentence in the main body of a webpage that contains keywords. If multiple keywords are contained, the keyword with the higher weight is used. Keywords here are words with higher weight values ​​obtained through weight calculation. When a sentence contains multiple keywords, using the keyword with the higher weight value is to broaden the distribution range of the long sentence, thereby improving its representativeness. This requires using the TF-IDF algorithm to calculate the weight of each word in the webpage's main body after word segmentation. The formulas for TF and IDF are:

[0022]

[0023] When the word n tWhen the number of occurrences in the webpage text is 18, and the total number of words in the webpage text, n, is 1000, its TF... t The value is 0.018, so word n t The weight value of the webpage content is 0.018.

[0024] When the word n t When the number of occurrences in the webpage text is 10, and the total number of words in the webpage text, n, is 100, its TF... t The value is 0.1, so word n t The weight value of the webpage content is 0.1.

[0025]

[0026] Where N represents the total number of documents in the corpus, N t Let t be the number of documents containing the word t in the corpus. In the TF-IDF algorithm, the importance of a word increases proportionally to the number of times it appears in a document, and decreases inversely proportionally to its frequency in the corpus. Therefore, it can effectively reflect the importance of words in an article and is computationally fast. Although the keywords obtained using the TF-IDF algorithm can represent the information in the text to a certain extent, due to the isolation of words and the existence of fixed collocations, the same word has a high probability of appearing in documents with the same topic, which may lead to dissimilar web pages being misclassified as duplicate pages. Therefore, if the context is incorporated and the length of the phrases combined with the keywords is increased, that is, feature extraction is performed on the parts of the paragraph with distinctiveness, the represented text features will be more obvious.

[0027] S5. Sentence segmentation (short and long sentences)

[0028] First, the main text structure is hierarchically divided based on the main text content. The main text structure is defined as a binary tuple (Root, F), where Root is the root node and F is a forest of m subtrees, F = (CT1, CT2, ..., CT...). n ), CT i =(r i F i The i-th subtree is called the root. Each node in the tree can be represented as a tuple TNode = (SID, SW), where SID is the number of the natural segment and SW is the weight of the natural segment. A long sentence is defined as a sentence with a character length of not less than k (usually k = 10 in experiments). The choice of length is very important. Too long a sentence will make the features susceptible to interference, while too short a sentence will result in poor representation of the features. For longer paragraphs, in order to reduce storage space, the longest m sentences (usually m = 4 in experiments) are selected. For paragraphs with less than m sentences, all sentences should be retained.

[0029] S6. Similarity calculation based on text length pre-classification

[0030] Dividing web pages into different sets based on their text length allows us to determine web page similarity only by comparing pages within sets of similar length, rather than comparing them to all web pages. The first step is to find the basis for this set division. The most intuitive approach is to divide all web pages in the database into several subsets based on their text length. Then, we can use the similarity formula between two web page documents d1 and d2 defined in the Shingling deduplication algorithm:

[0031] If we assume |A| < |B|, then the above formula can be scaled as follows:

[0032]

[0033] The similarity between two web page documents is linked to the length of the document body. If the threshold for judging similarity is set to t, then the two documents are only judged to be similar when sim(d1, d2) ≥ t, that is, |d2| ≤ |d1| / t.

[0034] S7. Repeated and Similar Processing

[0035] By combining webpage feature extraction based on key long sentences with similarity calculation based on pre-classification of text length, duplicate similar webpages on mobile devices can be processed, such as withdrawing duplicate similar webpages.

[0036] Preferably, in step S3, L0 is the link length of the sample text.

[0037] Preferably, in step S4, n t This indicates the number of times the word 't' appears in the main text of the webpage.

[0038] Preferably, in step S4, N represents the total number of documents in the corpus.

[0039] Preferably, in step S6, A and B are Shingling sets of length k extracted from documents d1 and d2, respectively, and can be transformed to obtain |d1|=|A|+k-1.

[0040] Preferably, in step S4, N t This represents the number of documents in the corpus containing the word t.

[0041] Preferably, in step S4, n represents the total number of words in the main text of the webpage.

[0042] Preferably, in step S6, for any given webpage document, once the similarity threshold is determined, the range of document text lengths that may be similar to that document can be determined. When comparing document similarity, the document text length space can be divided into several intervals according to the set upper limit of document text length and the similarity threshold. Any given document will fall into one of these intervals, while documents similar to that document can only fall into the same length interval. At the same time, documents that are not in the length interval will be filtered out.

[0043] The beneficial effects of this invention are as follows: Compared with the prior art, this invention provides a webpage processing method based on pre-classification of key long sentences and text length, which has the following beneficial effects:

[0044] 1. This webpage processing method based on key long sentences and text length pre-classification combines the advantages of webpage feature code extraction based on key long sentences and semantic feature extraction techniques. It also leverages the importance of semantic key long sentences and the connections between contexts, reducing the number of required features and greatly improving the accuracy of feature extraction and deduplication efficiency.

[0045] 2. This webpage processing method based on key long sentences and text length pre-classification, through a webpage text information extraction method based on Chinese punctuation mark HTML tree structure, has strong versatility and high accuracy. It can be used as a preprocessing method for search engines, webpage classification, information extraction, information filtering, information reconstruction and topic information collection, etc., greatly improving the efficiency of semantic cohesive text processing of webpages. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a schematic diagram of the processing flow of the present invention;

[0048] Figure 2 This is a diagram of an HTML tree structure. Detailed Implementation

[0049] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, the present invention provides a technical solution:

[0051] A webpage processing method based on pre-classification of key long sentences and body text length includes the following steps:

[0052] S1. Extraction of main text information

[0053] First, the entire webpage data needs to be processed to extract the HTML tree structure. The webpage data source is selected from 10 different portal websites, such as Sina, NetEase, and Sohu. Ten text-based webpages are randomly selected from each website, so that the entire content of the text is extracted from the HTML tree. The text includes the topic and content of the text. The HTML tree is traversed to extract all <text> nodes, and the link from the root node to each <text> leaf node is recorded. Here, a recursive algorithm is used to traverse the HTML tree.

[0054] S2. Determination of Text Structure Samples

[0055] The system checks the number of Chinese periods in each text segment and selects the link corresponding to the text segment with the largest number of periods as the structural sample. The text segment with the largest number of periods refers to the text segment containing the most periods in the selected text. Other text segments with a large number of Chinese periods are selected and their link structures are compared with the structural sample to determine the correctness of the structural sample.

[0056] S3, Webpage Feature Selection Threshold

[0057] For the main text content, the similarity threshold is chosen primarily considering frequently occurring links and modified text blocks within the content. Taking into account the worst-case scenario where some text blocks are simultaneously linked and modified, the number of nodes whose link structure matches the sample link structure is reduced by 1, and the link length is increased by 2, resulting in the following similarity:

[0058] Where L0 is the link length of the sample text, and K0 is the index that measures the similarity between link i and link j.

[0059] For example: when the link length of the L0 sample text is 11,

[0060] K0 is calculated using the above formula, and K0 = 0.83, so the similarity between link i and link j is 0.83.

[0061] When the link length of the L0 sample text is 18,

[0062] K0 is calculated using the above formula, and K0 = 0.89, so the similarity between link i and link j is 0.89.

[0063] When the link length of the L0 sample text is 4,

[0064] K0 is calculated using the above formula, and K0 = 0.6, so the similarity between link i and link j is 0.6.

[0065] S4. Webpage Feature Extraction Based on Key Long Sentences

[0066] A key long sentence refers to the longest sentence in the main body of a webpage that contains keywords. If multiple keywords are contained, the keyword with the higher weight is used. Keywords here are words with higher weight values ​​obtained through weight calculation. When a sentence contains multiple keywords, using the keyword with the higher weight value is to broaden the distribution range of the long sentence, thereby improving its representativeness. This requires using the TF-IDF algorithm to calculate the weight of each word in the webpage's main body after word segmentation. The formulas for TF and IDF are:

[0067] Where, n t This indicates the number of times word 't' appears in the main text of the webpage, where 'n' represents the total number of words in the main text of the webpage.

[0068] When the word n t When the number of occurrences in the webpage text is 18, and the total number of words in the webpage text, n, is 1000, its TF... t The value is 0.018, so word n t The weight value of the webpage content is 0.018.

[0069] When the word n t When the number of occurrences in the webpage text is 10, and the total number of words in the webpage text, n, is 100, its TF... t The value is 0.1, so word n t The weight value of the webpage content is 0.1.

[0070] Where N represents the total number of documents in the corpus, N t This represents the number of documents in the corpus containing the word t.

[0071] In the TF-IDF algorithm, the importance of a word increases proportionally to its frequency of occurrence in a document and decreases inversely proportionally to its frequency of occurrence in the corpus. Therefore, it can effectively reflect the importance of words in an article and is computationally fast. While the keywords obtained using the TF-IDF algorithm can represent the information in the text to some extent, the isolation of words and their fixed collocations mean that the same words appear more frequently in documents with the same topic, potentially misclassifying dissimilar web pages as duplicates. Therefore, by incorporating contextual content and increasing the length of phrases combined with keywords—that is, extracting features from the distinctive parts of paragraphs—the resulting text features will be more clearly represented.

[0072] S5. Sentence segmentation (short and long sentences)

[0073] First, the main text structure is hierarchically divided based on the main text content. The main text structure is defined as a binary tuple (Root, F), where Root is the root node and F is a forest of m subtrees, F = (CT1, CT2, ..., CT...). n ), CT i =(r i F i The i-th subtree is called the root. Each node in the tree can be represented as a tuple TNode = (SID, SW), where SID is the number of the natural segment and SW is the weight of the natural segment. A long sentence is defined as a sentence with a character length of not less than k (usually k = 10 in experiments). The choice of length is very important. Too long a sentence will make the features susceptible to interference, while too short a sentence will result in poor representation of the features. For longer paragraphs, in order to reduce storage space, the longest m sentences (usually m = 4 in experiments) are selected. For paragraphs with less than m sentences, all sentences should be retained.

[0074] S6. Similarity calculation based on text length pre-classification

[0075] Dividing web pages into different sets based on their text length allows us to determine web page similarity only by comparing pages within sets of similar length, rather than comparing them to all web pages. The first step is to find the basis for this set division. The most intuitive approach is to divide all web pages in the database into several subsets based on their text length. Then, we can use the similarity formula between two web page documents d1 and d2 defined in the Shingling deduplication algorithm:

[0076] If we assume |A| < |B|, then the above formula can be scaled as follows:

[0077]

[0078] A and B are shingling sets of length k extracted from documents d1 and d2, respectively. A transformation yields |d1| = |A| + k - 1. This links the similarity of two webpage documents to the length of their text. If the similarity threshold is set to t, then the two documents are considered similar only when sim(d1, d2) ≥ t, i.e., |d2| ≤ |d1| / t. For any given webpage document, once the similarity threshold is determined, the range of text lengths of documents potentially similar to that document can be identified. When comparing document similarity, the text length space can be divided into several intervals based on the set upper limit of the text length and the similarity threshold. Any given document will fall within one of these intervals, while documents similar to it can only fall within the same length interval. Documents outside this length interval will be filtered out.

[0079] S7. Repeated and Similar Processing

[0080] By combining webpage feature extraction based on key long sentences with similarity calculation based on pre-classification of text length, duplicate similar webpages on mobile devices can be processed, such as withdrawing duplicate similar webpages.

[0081]

[0082]

[0083] Note: Precision = Number of correctly detected duplicate web pages / Number of detected duplicate web pages; Recall = Number of correctly detected duplicate web pages / Number of actual duplicate web pages.

[0084] The beneficial effects of this invention are: by combining the advantages of webpage feature code extraction based on key long sentences and semantic feature extraction techniques, while leveraging the importance of semantic key long sentences and the connections between contexts, the number of required features is reduced, greatly improving the accuracy of feature extraction and the efficiency of deduplication; the webpage text information extraction method based on Chinese punctuation HTML tree structure is highly versatile and accurate, and can be used as a preprocessing method for search engines, webpage classification, information extraction, information filtering, information reconstruction, and topic information collection, greatly improving the efficiency of semantic cohesive text processing of webpages.

[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A webpage processing method based on pre-classification of key long sentences and body text length, characterized in that, Includes the following steps: S1. Extraction of main text information First, the entire webpage data needs to be processed to extract the HTML tree structure. The webpage data comes from multiple different portal websites. Multiple text-based webpages are randomly selected from each website so that the entire content of the text in the HTML tree can be extracted. The text includes the topic and content of the text. The HTML tree is traversed to extract all text nodes, and the link from the root node to each text leaf node is recorded. Here, a recursive algorithm is used to traverse the HTML tree recursive function. S2. Determination of Text Structure Samples Check the number of Chinese periods in each text segment and select the link corresponding to the text segment with the largest number of periods as the structural sample. The text segment with the largest number of periods refers to the text segment with the most periods in the selected text. Select other text segments with a large number of Chinese periods and compare their link structure with the structural sample to determine the correctness of the structural sample. S3, Webpage Feature Selection Threshold For the main text content, the similarity threshold is chosen by considering frequently occurring links and modified text blocks within the main text. When a portion of the text block is both linked and modified, the number of nodes whose link structure matches the sample link structure decreases by 1, and the link length increases by 2, resulting in the following similarity: Where L0 is the link length of the sample text, and K0 is the index that measures the similarity between link i and link j; S4. Webpage Feature Extraction Based on Key Long Sentences A key long sentence refers to the longest sentence in the main body of a webpage that contains keywords. If multiple keywords are contained, the keyword with the higher weight is used. Keywords are words with higher weight values ​​obtained through weight calculation. The TF-IDF algorithm is used to calculate the weight of each word in the main body of the webpage after word segmentation. The formulas for calculating TF and IDF are as follows: ， ； Where n represents the total number of words in the main text of the webpage, n t N represents the number of times word t appears in the main text of the webpage, N represents the total number of documents in the corpus, and Nt is the number of documents in the corpus containing word t. S5. Sentence segmentation (short and long sentences) First, the text structure is hierarchically divided based on the main content, and defined as a binary tuple (Root, F), where Root is the root node, and F is a forest of m subtrees, F = (CT1, CT2, ..., CT...). n ), CT i = (r i F i The i-th subtree is called the root. Each node in the tree is represented as a tuple TNode = (SID, SW), where SID is the number of the natural segment and SW is the weight of the natural segment. Then, a long sentence is defined as having a character length of no less than k. The m longest sentences among all long sentences are selected. For paragraphs with fewer than m sentences, all sentences should be retained. Then, the summary of all long sentences is calculated, and the vector composed of these summaries is used as the feature of the paragraph. When determining whether a paragraph is a duplicate paragraph, the number of sentences contained in the paragraph should be calculated first, and then the long sentence extraction algorithm should be used to find the feature of the paragraph. S6. Similarity calculation based on text length pre-classification By dividing web pages into different sets based on their text length, when determining web page similarity, it's only necessary to compare web pages within sets with similar text lengths, rather than comparing them with all web pages. First, the basis for set division should be found: divide all web pages in the database into several subsets based on their text length. Then, use the similarity formula between two web page documents d1 and d2 defined in the Shingling deduplication algorithm: If we assume that |A| < |B|, then the above formula is scaled as follows: ， The similarity between two web page documents is linked to the length of the document body. If the threshold for judging similarity is set to t, then the two documents are only judged to be similar when sim(d1,d2)≥t, that is, |d2|≤|d1| / t. S7. Repeated and Similar Processing By combining webpage feature extraction based on key long sentences with similarity calculation based on pre-classification of text length, duplicate similar webpages on mobile devices can be processed.

2. The webpage processing method based on key long sentences and text length pre-classification according to claim 1, characterized in that, In step S6, A and B are Shingling sets of length k extracted from documents d1 and d2, respectively, and are transformed to obtain |d1| = |A| + k - 1.

Images