A multi-angle feature learning-based topic web information extraction method

Abstract

The application relates to the technical field of webpage information extraction, and specifically discloses a theme webpage information extraction method based on multi-angle feature learning, and the application uses a webpage crawler to acquire the HTML source code of a page, can extract the content features of the page, and includes a title, a domain name, an IP belonging place and the like.For the features in the webpage title and webpage content, corresponding string matching and recognition algorithms are adopted to realize feature extraction; for domain name information, IP belonging place, search engine and the like, corresponding interfaces are used to complete information extraction and feature acquisition; the theme webpage information extraction method based on multi-angle feature learning improves the F value of the keyword method by more than 4%, thereby verifying the effectiveness of the method; compared with the SVM method, the F value of the keyword method is reduced by 10%; compared with the method in the application, the F value is reduced by 17%, so the method is superior to other methods in terms of algorithm feasibility and effect.

Description

Technical Field

[0001] This invention relates to the field of web page information extraction technology, specifically to a method for extracting topic web page information based on multi-angle feature learning. Background Technology

[0002] The ability to quickly extract key information from the vast amount of online data is crucial not only for online ecosystem governance but also for timely understanding of public opinion trends. The backflow and dissemination of specific information can negatively impact public perception and lead to misconceptions.

[0003] Current methods for discovering web page information suffer from difficulties, slowness, and inaccuracy. The paper "A Review of Deep Learning-Based Methods for Extracting Social Network Public Opinion Information" proposes a deep learning-based method for extracting social network public opinion information, but its applicability is limited. The paper "A Fast Algorithm for Extracting Big Data Information Based on Artificial Intelligence" proposes an AI-based algorithm for extracting big data information, but it lacks a quantitative and accurate description of data standardization. Therefore, accurate extraction of web page topic information is particularly necessary. Summary of the Invention

[0004] The purpose of this invention is to provide a method for extracting topic web page information based on multi-angle feature learning, so as to solve the problems mentioned in the background art.

[0005] Technical solution

[0006] This invention provides the following technical solution: a method for extracting topic webpage information based on multi-angle feature learning, including the principles of topic webpage extraction and web crawler extraction.

[0007] A method for extracting topic webpage information based on multi-angle feature learning includes the following steps:

[0008] 1. Extraction of thematic web pages;

[0009] 2. Web crawler extraction principle.

[0010] (1) Extraction of thematic web pages

[0011] Web extraction is the process of extracting meaningful information from web pages. Common methods involve using the structural, textual, or domain-specific information of the web page to determine which parts of the content are more important.

[0012] Structural information includes the contextual relationships between text, the hierarchical relationships between nodes in the DOM tree, and the visual positional relationships of web page elements. Textual information includes linguistic characteristics such as the degree of aggregation between text blocks. Domain information is relative to a specific application. This paper studies topic-oriented web page information extraction. A topic is information related to the target needs.

[0013] (2) Web crawler extraction principle

[0014] Web topic text filtering requires page analysis, which is performed using a web crawler. The web crawler automatically retrieves web pages from the internet according to certain rules. It crawls pages from the initial URL, obtains new URLs, downloads and parses the pages, checks for duplicate content, filters and removes duplicate URLs, and adds the new URLs to a URL queue. This process continues until a set stopping condition is met, at which point the crawler stops.

[0015] A method for extracting topic webpage information based on multi-angle feature learning includes the following steps:

[0016] I. Text segmentation

[0017] II. Multi-angle feature analysis

[0018] Seven features were selected from the perspectives of webpage title, webpage content, domain information, and IP address location. Thematic features of the webpages were extracted, quantified, and feature vectors were generated for analysis.

[0019] The webpage title and webpage content are analyzed separately, and corresponding weights are assigned to their respective features. The specific process is as follows:

[0020] Using the characteristic function F i To characterize their corresponding identity features, define the identity feature vector for a given webpage:

[0021] F i ={F1, F2, F3, F4, F5, F6, F7} (1)

[0022] Among them, F1 indicates whether the webpage title contains the main keywords of the topic, F2 indicates whether the webpage title hits the related keywords of the topic, F3 indicates whether the webpage content contains the main keywords of the topic, F4 indicates whether the webpage content hits the related keywords of the topic, F5 indicates whether the domain registrar is an overseas company, F6 indicates whether the IP address is located overseas, and F7 indicates whether the webpage title returns results in the search engine.

[0023] The output of each feature function is a real value, representing the corresponding identity feature status on the webpage, as defined below:

[0024] F1: Does the webpage title contain key words of the theme? To highlight the key points, webpages often use eye-catching titles to attract readers. Such titles often contain key words of the theme.

[0025] The definition of F1 is as follows:

[0026] ;

[0027] F2: Does the webpage title match the topic's related keywords?

[0028] F2 is defined as follows:

[0029] ;

[0030] F3: Does the webpage content contain key keywords related to the topic?

[0031] F3 is defined as follows:

[0032] ;

[0033] F4: Check if the webpage content contains specific sensitive words;

[0034] F4 is defined as follows:

[0035] ;

[0036] F5: Is the domain registrar an overseas company?

[0037] F5 is defined as follows:

[0038] ;

[0039] F6: Is the IP address located outside of China?

[0040] F6 is defined as follows:

[0041] ;

[0042] F7: The page title in the search engine results; F7 is defined as follows:

[0043] ;

[0044] III. Deep Learning Training of the Model

[0045] The above F1, F2...F i , The sensitive features are processed through independent recurrent neural networks, and then through fully connected neural networks for later fusion, so that each independent recurrent neural network can learn them, resulting in a high model accuracy.

[0046] Specifically, based on the above characteristics, when determining whether a website is a topic webpage, a linear classifier is used for processing. The linear classification function is shown in equation (2):

[0047] (2)

[0048] In equation (2), F i F represents the value of the sensitive identity feature of the webpage. i When =1, the website is determined to be a topic website; Fi When =-1, it indicates that the page is normal and the website is a normal website. In formula (2), w i The weights for the seven sensitive feature components are calculated using the formula (3):

[0049] (3)

[0050] In the above formula, e i The calculation formula is equation (4):

[0051] (4)

[0052] In equation (4), and These represent the accuracy and false positive rate when using the sensitive feature Fi alone to detect pages;

[0053] Using this method, the resulting S value will be in the range [-1, 1]. When S is close to -1, the webpage is considered a non-topic webpage, and when the S value is closer to 1, the webpage is considered a topical webpage.

[0054] Preferably, the use of web crawlers to obtain the HTML source code of the page allows for the extraction of page content features, including title, domain name, and IP address. For features in the page title and content, appropriate string matching and recognition algorithms are used to extract features; for data such as domain name information, IP address, and search engine information, corresponding interfaces are used to complete information extraction and feature acquisition.

[0055] Preferably, the equation (2) is a linear function with 7 elements, namely 7 influencing factors. These 7 factors correspond to the values ​​of 7 features. At the same time, the introduction of weight coefficients is to quantify the proportion of the influence of each feature. The larger the weight coefficient, the greater the influence of this feature on the result in this linear function.

[0056] Beneficial effects

[0057] Compared with existing technologies, this invention provides a method for extracting topic webpage information based on multi-angle feature learning, which has the following beneficial effects:

[0058] 1. The proposed topic webpage information extraction method based on multi-angle feature learning improves the F-value by more than 4% compared to the keyword method, verifying the effectiveness of the method. Compared to the SVM method, the keyword method reduces the F-value by 10%. Compared to the proposed method, the F-value is reduced by 17%. It can be seen that the proposed method is superior to other methods in terms of both algorithm feasibility and performance. Attached Figure Description

[0059] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0060] Figure 1 This is the main workflow of the multi-angle feature analysis of this invention;

[0061] Figure 2 This is the main principle of the crawler extraction in this invention;

[0062] Figure 3 This is the hierarchical structure of the deep learning model of the present invention;

[0063] Figure 4 This is a comparison of the F-values ​​of different methods in this invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example:

[0066] Please see Figures 1-4 This invention provides a technical solution: a method for extracting topic webpage information based on multi-angle feature learning, including the principles of topic webpage extraction and web crawler extraction.

[0067] Web extraction is the process of extracting meaningful information from web pages. Common methods include using the structural, textual, or domain-specific information of the web page to determine which parts of the content are more important.

[0068] Structural information includes the contextual relationships between text, the hierarchical relationships between nodes in the DOM tree, and the visual positional relationships of web page elements. Textual information includes linguistic characteristics such as the degree of aggregation between text blocks. Domain information is relative to a specific application; this paper studies topic-oriented web page information extraction. A topic is information related to the target needs.

[0069] Web crawler extraction principle:

[0070] Web-based topic text filtering requires page analysis, which is performed using a web crawler. The web crawler automatically retrieves web pages from the internet according to certain rules. Starting with an initial URL, it crawls pages to obtain new URLs, downloads and parses the pages, checks for duplicate content, filters and removes duplicate URLs, and adds the new URLs to a URL queue. This process continues until a set stopping condition is met, at which point the crawler stops.

[0071] A method for extracting topic webpage information based on multi-angle feature learning includes the following steps:

[0072] I. Text segmentation

[0073] II. Multi-angle feature analysis

[0074] Seven features were selected from the perspectives of webpage title, webpage content, domain information, and IP address location. Thematic features of the webpages were extracted, quantified, and feature vectors were generated for analysis.

[0075] The webpage title and webpage content are analyzed separately, and corresponding weights are assigned to their respective features. The specific process is as follows: using the feature function F... i To characterize their corresponding identity features, define the identity feature vector for a given webpage:

[0076] F i ={F1, F2, F3, F4, F5, F6, F7} (1)

[0077] Among them, F1 indicates whether the webpage title contains the main keywords of the topic, F2 indicates whether the webpage title hits the related keywords of the topic, F3 indicates whether the webpage content contains the main keywords of the topic, F4 indicates whether the webpage content hits the related keywords of the topic, F5 indicates whether the domain registrar is an overseas company, F6 indicates whether the IP address is located overseas, and F7 indicates whether the webpage title returns results in the search engine.

[0078] The output of each feature function is a real value, representing the corresponding identity feature status on the webpage, as defined below:

[0079] F1: Does the webpage title contain key words of the theme? To highlight the key points, webpages often use eye-catching titles to attract readers. Such titles often contain key words of the theme.

[0080] The definition of F1 is as follows:

[0081] ;

[0082] F2: Does the webpage title match the topic's related keywords?

[0083] F2 is defined as follows:

[0084] ;

[0085] F3: Does the webpage content contain key keywords related to the topic?

[0086] F3 is defined as follows:

[0087] ;

[0088] F4: Check if the webpage content contains specific sensitive words;

[0089] F4 is defined as follows:

[0090] ;

[0091] F5: Is the domain registrar an overseas company?

[0092] F5 is defined as follows:

[0093] ;

[0094] F6: Is the IP address located outside of China?

[0095] F6 is defined as follows:

[0096] ;

[0097] F7: The page title in the search engine results; F7 is defined as follows:

[0098] ;

[0099] III. Deep Learning Training of the Model

[0100] The above F1, F2...F i , The sensitive features are processed through independent recurrent neural networks, and then through fully connected neural networks for later fusion, so that each independent recurrent neural network can learn them, resulting in a high model accuracy.

[0101] Specifically, based on the above characteristics, when determining whether a website is a topic webpage, a linear classifier is used. The linear classification function is shown in equation (2):

[0102] (2)

[0103] In equation (2), F i F represents the value of the sensitive identity feature of the webpage. i When =1, the website is determined to be a topic website; F i When =-1, it indicates that the page is normal and the website is a normal website. In formula (2), w iThe weights for the seven sensitive feature components are calculated using the formula (3):

[0104] (3)

[0105] In the above formula, e i The calculation formula is equation (4):

[0106] (4)

[0107] In equation (4), and These represent the accuracy and false positive rate when using the sensitive feature Fi alone to detect pages;

[0108] Using this method, the resulting S value will be in the range [-1, 1]. When S is close to -1, the webpage is considered a non-topic webpage, and when the S value is closer to 1, the webpage is considered a topical webpage.

[0109] In this embodiment, web crawlers are used to obtain the HTML source code of the page, allowing for the extraction of page content features, including title, domain name, and IP address. For features in the page title and content, appropriate string matching and recognition algorithms are used to extract features; for data such as domain name information, IP address, and search engine information, corresponding interfaces are used to complete information extraction and feature acquisition.

[0110] In this embodiment, equation (2) is a linear function with 7 elements, namely 7 influencing factors. These 7 factors correspond to the values ​​of 7 features. At the same time, the introduction of weight coefficients is to quantify the proportion of the influence of each feature. The larger the weight coefficient, the greater the influence of this feature on the result in this linear function.

[0111] The working principle of this embodiment is as follows: Based on specific keywords, web page information is collected on a large scale through Internet crawlers, including web page domain name, URL, title, content and other information. The collected web page information is centrally stored. The experiment mainly focuses on various types of text information, because images, audio and video information in the web page content are not crawled and stored.

[0112] A certain percentage of web pages are selected for page feature analysis to summarize patterns and characteristics.

[0113] Based on the web page features extracted automatically by the program, and on the basis of labeled web page data, the model is trained through deep learning. The model is trained using a fully connected neural network deep learning model, and a linear classifier is used to classify the web pages.

Claims

1. A method for extracting topic webpage information based on multi-angle feature learning, characterized in that: This includes the principles of topic webpage extraction and web crawler extraction; A method for extracting topic webpage information based on multi-angle feature learning includes the following steps: I. Text segmentation II. Multi-angle feature analysis We selected seven features from the perspectives of webpage title, webpage content, domain name information, and IP address location. By extracting the thematic features of the webpage, we quantified them and generated feature vectors for analysis. The webpage title and webpage content are analyzed separately, and corresponding weights are assigned to their respective features. The specific process is as follows: A feature function Fi is used to characterize the corresponding identity features, defining the identity feature vector for a given webpage: F i ={F1, F2, F3, F4, F5, F6, F7}(1) Among them, F1 indicates whether the webpage title contains the main keywords of the topic, F2 indicates whether the webpage title hits the related keywords of the topic, F3 indicates whether the webpage content contains the main keywords of the topic, F4 indicates whether the webpage content hits the related keywords of the topic, F5 indicates whether the domain registrar is an overseas company, F6 indicates whether the IP address is located overseas, and F7 indicates whether the webpage title returns results in the search engine. The output of each feature function is a real value, representing the corresponding identity feature status on the webpage, as defined below: F1: Does the webpage title contain key keywords related to the topic? The definition of F1 is as follows: ； F2: Does the webpage title match the topic's related keywords? F2 is defined as follows: ； F3: Does the webpage content contain key keywords related to the topic? F3 is defined as follows: ； F4: Check if the webpage content contains specific sensitive words; F4 is defined as follows: ； F5: Is the domain registrar an overseas company? F5 is defined as follows: ； F6: Is the IP address located outside of China? F6 is defined as follows: ； F7: Page title in search engine results; F7 is defined as follows: ； The above F1, F2...F i , The sensitive features are processed through independent recurrent neural networks, and then through fully connected neural networks for later fusion, so that each independent recurrent neural network can learn them. Specifically, based on the above characteristics, when determining whether a website is a topic webpage, a linear classifier is used for processing. The linear classification function is shown in equation (2): （2） In equation (2), F i F represents the value of the sensitive identity feature of the webpage. i When =1, the website is determined to be a topic website; F i When =-1, it indicates that the page is normal and the website is a normal website. In formula (2), w i The weights for the seven sensitive feature components are calculated using the formula (3): （3） In the above formula, e i The calculation formula is equation (4): （4） In equation (4), and Using sensitive feature F separately i Accuracy and false positive rates when detecting web pages; Using this method, the resulting S value will be in the range [-1, 1]. When S is close to -1, the webpage is considered a non-topic webpage, and when the S value is closer to 1, the webpage is considered a topical webpage.

2. The method for extracting topic webpage information based on multi-angle feature learning according to claim 1, characterized in that: The method of using web crawlers to obtain the HTML source code of a page allows for the extraction of page content features, including title, domain name, and IP address. For features in the page title and content, corresponding string matching and recognition algorithms are used to extract features. For domain name information, IP address, and data from search engines, corresponding interfaces are used to complete information extraction and feature acquisition.

3. The method for extracting topic webpage information based on multi-angle feature learning according to claim 1, characterized in that: Equation (2) is a linear function with 7 elements, namely 7 influencing factors. These 7 factors correspond to the values ​​of 7 features. At the same time, the introduction of weight coefficients is to quantify the proportion of the influence of each feature. The larger the weight coefficient, the greater the influence of this feature on the result in this linear function.

Images