Method for automatic extraction and structure restoration of privacy policy for online service
By crawling privacy policy HTML documents and using machine learning classifiers to identify heading levels, this method solves the problem of insufficient webpage structure reconstruction in existing technologies, achieving high-accuracy structure reconstruction and chapter division, and supporting privacy policy analysis for online services.
Patent Information
- Application Number
- CN202210799880.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-06
- Publication Date
- 2026-02-24
- Estimated Expiration
- 2042-07-06
AI Technical Summary
Existing automated webpage analysis technologies are unable to deeply analyze the hierarchical structure of titles and text, especially for privacy policy webpages, where structural reconstruction is insufficient.
By locating the privacy policy link, the original HTML document is crawled, and the main body of the privacy policy is identified using a method based on the uniformity of node text distribution. A machine learning classifier is then used to identify the title and hierarchy, and the XML file structure is reconstructed.
It achieves an accuracy rate of over 85% in recognizing multi-level headings on privacy policy webpages, effectively restoring their hierarchical structure and dividing them into chapters, and supporting compliance analysis of privacy policies for online services.
Smart Images
Figure CN115292638B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a technology in the field of automated analysis applications, specifically a method for automatically extracting and restoring the structure of privacy policies for online services. Background Technology
[0002] Existing automated webpage analysis technologies typically analyze the underlying HTML document structure of webpages, using HTML tags to build text classifiers for paragraph and heading identification. However, current methods only simply identify headings and paragraph text without conducting in-depth analysis to reconstruct the hierarchical structure information of the headings. Summary of the Invention
[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a method for automatically extracting and reconstructing the structure of privacy policies for online services, specifically targeting privacy policies as a particular type of webpage. This method enables automated analysis and reconstruction of privacy policies in online services, providing support for compliance and integrity analysis of privacy policies in online services.
[0004] This invention is achieved through the following technical solution:
[0005] This invention relates to a method for automatically extracting and restoring the structure of privacy policies for online services. In an online service website, the method locates a privacy policy link and jumps to the corresponding privacy policy webpage. The original HTML (Hypertext Markup Language) document of the privacy policy webpage is crawled through a plugin. After title recognition and title hierarchy recognition, the hierarchical structure of the privacy policy is finally restored.
[0006] The aforementioned method for locating privacy policy links involves using a script to access the URLs (Uniform Resource Locators) of online service websites that rank highly in Alexa's website rankings, and then using keyword searches to find the location of the privacy policy and the corresponding links within those websites.
[0007] The aforementioned plugin crawling method accesses the privacy policy webpage of the online service via a script, uses the SingleFile plugin to crawl the original HTML file of the privacy policy webpage, and obtains the HTML document of the online service's privacy policy.
[0008] The aforementioned title recognition and title hierarchy recognition, targeting the privacy policy HTML document, identifies all HTML nodes in the main body of the privacy policy and their smallest parent node using a text extraction method based on the uniformity of node text distribution; filters the node text to obtain all title nodes in the privacy policy, and uses a title hierarchy classifier to obtain the hierarchy to which each title belongs; finally, the structure of the entire privacy policy is restored and stored as an XML (eXtensible Markup Language) file.
[0009] This invention relates to an automatic extraction and structure restoration system for privacy policies of online services that implements the above-mentioned method. The system includes: a privacy policy crawling module, a privacy policy subject identification module, and a privacy policy structure restoration module. Specifically: the privacy policy crawling module locates privacy policies in online service websites using keywords and obtains the original HTML file of the privacy policy webpage using the SingleFile plugin; the privacy policy subject identification module determines the location of the privacy policy subject in the HTML file based on the text distribution patterns of privacy policy nodes and extracts all nodes of the privacy policy subject in the HTML file; the privacy policy structure restoration module extracts node attributes such as the style of the first sequence number tag, text length, parent node tag name, font size, font weight, node depth, whether italicized, and whether it has an underline as features, and uses machine learning to build a classifier to identify whether each node is a title and the title's hierarchy.
[0010] The privacy policy crawling module includes a privacy policy locating unit and a privacy policy crawling unit. The privacy policy locating unit locates the location and link of an online service's privacy policy by searching for keywords such as privacy policy and privacy terms on the online service website, and then jumps to the privacy policy webpage of the online service. The privacy policy crawling unit uses the SingleFile plugin to crawl the original HTML file of the privacy policy webpage to obtain the HTML document of the online service's privacy policy.
[0011] The privacy policy body identification module recursively finds the node containing the body of the privacy policy in the privacy policy HTML document by extracting the body text based on the uniformity of node text distribution. That is, it finds the smallest parent node containing all the body content in the HTML DOM tree.
[0012] The privacy policy structure restoration module extracts node attributes such as the style of the first serial number tag, text length, parent node tag name, font size, font weight, node depth, whether it is italic, and whether it has an underline as features, and uses machine learning to build a classifier to identify whether each node is a title and the title level.
[0013] Technical effect
[0014] This invention uses algorithms to locate the main body of a webpage and employs machine learning to distinguish between heading elements and paragraph elements. It extracts the main content of an HTML webpage containing a privacy policy and differentiates various elements, ultimately restoring the hierarchical structure of the privacy policy. Attached Figure Description
[0015] Figure 1 This is a diagram of the overall architecture of the present invention. Detailed Implementation
[0016] like Figure 1 As shown in this embodiment, an automatic extraction and structure restoration system for privacy policies of online services includes: a privacy policy crawling module, a privacy policy subject identification module, and a privacy policy structure restoration module. The privacy policy crawling module accesses the URLs (Uniform Resource Locators) of online service websites ranking high in Alexa search results via scripts, locates the privacy policies and corresponding links on the websites using keyword searches, accesses the privacy policy webpage of the online service via scripts, and uses the SingleFile plugin to crawl the original HTML file of the privacy policy webpage, obtaining the HTML document of the privacy policy of the online service. The privacy policy subject identification module takes the HTML document of the privacy policy output by the privacy policy crawling module as input and identifies the smallest parent node of the main body of the privacy policy using a text extraction method based on the uniformity of node text distribution. The privacy policy structure restoration module takes all HTML nodes of the main body of the privacy policy output by the privacy policy subject identification module as input, filters the node text by constructing filters to obtain all title nodes in the privacy policy, obtains the level to which each title belongs using a title hierarchy classifier, and finally restores the structure of the entire privacy policy and stores it as an XML (eXtensible Markup Language) file.
[0017] The privacy policy crawling module includes a privacy policy location unit and a privacy policy crawling unit. The privacy policy location unit involves constructing an automated script to locate the privacy policy information and links on online service websites. It selects top-ranking online service websites from Alexa website rankings, obtaining their URLs to form a set of online service URLs. A Python script accesses each URL in this set, using keyword search to find keywords such as "privacy," "privacy policy," and "privacy terms" on the online service website, ultimately obtaining the location and link of the privacy policy within that website. The privacy policy crawling unit constructs an automated script to crawl the privacy policy information. A Python script accesses the privacy policy link obtained from the privacy policy location unit and uses the SingleFile plugin to save the privacy policy page to an HTML file. SingleFile is a browser extension that can save a complete page to an HTML file, including page tags, styles, and other information.
[0018] Based on the privacy policy crawling module mentioned above, a total of 150 online service websites' privacy policy HTML files were collected across 15 categories.
[0019] The privacy policy body identification module extracts the main body of the privacy policy in the HTML file based on the text extraction method of node text distribution uniformity. Node text refers to the content remaining after removing all HTML tags from a node, which can also be understood as the leaf node of the HTML DOM tree. Distribution refers to the text length distribution of all direct child nodes under the same parent node, which is calculated iteratively in the DOM tree. The uniformity is measured by the standard deviation of the length distribution. When the ratio of the standard deviation obtained in a certain iteration to the average of all non-zero standard deviations obtained in the previous iteration is lower than a preset threshold, the distribution is considered uniform.
[0020] The text extraction method based on the uniformity of node text distribution specifically includes:
[0021] Step 1: Initialize an empty list R to store the standard deviation calculated in each subsequent iteration; initialize the ratio p of the standard deviation of the current iteration to the average standard deviation of previous iterations to 1; initialize the target node to the body node to participate in the iteration.
[0022] Step 2: Iteratively generate a list C of child nodes for the target node, and calculate the corresponding standard deviation S based on the text length of each child node. When the p value is greater than 0.55 or S = 0, update the p value and add S to R, with the target node iterating to its longest-text-length child node; otherwise, the iteration ends, indicating that the target node has been found. This target node is the smallest parent node containing the main text of the privacy policy.
[0023] Based on the aforementioned privacy policy subject identification module, the privacy policy portion is extracted from the HTML file of each privacy policy webpage. This is equivalent to pruning the original HTML DOM tree, providing support for the subsequent restoration of the privacy policy structure. The extracted privacy policy subject is still an HTML DOM tree.
[0024] The privacy policy structure restoration module extracts node attributes such as the style of the first serial number tag, text length, parent node tag name, font size, font weight, node depth, whether it is italic, and whether it has an underline as features, and uses machine learning to build a classifier to identify whether each node is a title and the title level.
[0025] The classifier identification specifically includes:
[0026] Step 1: Extract the following 7 attributes of the current element, which may help distinguish between heading elements and paragraph elements: 1) Text length, 2) Font size relative to the body node's font base, 3) Font weight relative to the body node's font base, 4) Whether the text is italic, 5) Whether the text has an underline, 6) Node depth in the HTML DOM tree, and 7) HTML tag level defined as follows: 1-6 are... <h1>，…，< / h1> <h6>7 indicates an inline HTML element (e.g., ...). <strong>， <em>， ， ， ,as well as (And related highlighting styles), 8 represents any other HTML element. Furthermore, for each of the three attributes—font size, font weight, and node depth—all unique values of all remaining HTML elements are sorted as additional features. In addition to the above 10 features, a 12-dimensional vector is used to describe the first-order tag style of the current element. For each candidate heading / paragraph element, there is a 22-D feature vector. An ml-based classifier can be constructed to predict that candidate elements belong to one of the following five classes: Level ~ $i$ ($i = 1, 2, 3, 4$) for headings and paragraphs. Based on our overall observation of the privacy policy, we selected four levels of headings. This maximum number of levels can be easily changed within our framework if necessary.
[0027] Step 2: Based on the 22-D feature vector extracted in Step 1, a machine learning-based classifier is used to predict the category of the current element, which is a heading or paragraph of level one to four. This determines whether the current element is a heading and its heading level. Several mainstream machine learning algorithms were tested, with the classifier based on the Extra Trees algorithm showing the best performance.
[0028] The privacy policy structure reconstruction module described above identifies the titles and hierarchical levels of the main body nodes of the privacy policy, thereby reconstructing the entire hierarchical structure of the privacy policy. The results of title and hierarchical title recognition are shown in the table below:
[0029]
[0030] In the table, precision represents the probability that a sample is actually positive out of all samples predicted as positive. Recall represents the probability that a sample is predicted as positive out of all samples that are actually positive. Based on the identification results of the privacy policy structure reconstruction module, the privacy policy webpage can be effectively divided into chapters and its structure reconstructed. An XML file is used to store the structured privacy policy webpage, with the root node being `policy`. Chapters, titles, and paragraphs are represented by `segment`, `title`, and `paragraph` nodes, respectively, and their hierarchical relationships are reflected through the XML structure.
[0031] Compared with existing technologies, this method uses the Python programming language to complete automated scripts for privacy policy crawling, privacy policy subject identification, and privacy policy structure reconstruction. The scripts are started using shell commands, and the accuracy rate for identifying multi-level headings in privacy policy web pages reaches over 85%.
[0032] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention. < / em> < / strong> < / h6>
Claims
1. A method for automatically extracting and restoring the structure of privacy policies for online services, characterized in that, In online service websites, by locating the privacy policy link and jumping to the corresponding privacy policy page, the original HTML document of the privacy policy page is crawled through the plugin. After title recognition and title level recognition, the hierarchical structure of the privacy policy is finally restored. The aforementioned title recognition and title hierarchy recognition, for the privacy policy HTML document, identify all HTML nodes in the main body of the privacy policy and their smallest parent node by using a text extraction method based on the uniformity of node text distribution; filter the node text by constructing a filter to obtain all title nodes in the privacy policy, and obtain the hierarchy to which each title belongs by using a title hierarchy classifier; finally, restore the structure of the entire privacy policy and store it as an XML file. The text extraction method based on the uniformity of node text distribution specifically includes: Step 1: Initialize an empty list R to store the standard deviation calculated in each subsequent iteration; initialize p, the ratio of the standard deviation of the current iteration to the average standard deviation of previous iterations, to 1; initialize the target node as a body node to participate in the iteration. Step 2: Iteratively generate a list C of child nodes of the target node, and calculate the corresponding standard deviation S based on the text length of the child nodes. When the p value is greater than 0.55 or S = 0, update the p value and add S to R. The target node iterates to its longest child node. Otherwise, the iteration ends, indicating that the target node has been found. The target node is the smallest parent node containing the main text of the privacy policy.
2. The method for automatically extracting and restoring the structure of privacy policies for online services according to claim 1, characterized in that, The method for locating privacy policy links involves using a script to access the URLs of online service websites that rank highly in Alexa's website rankings, and then using keyword searches to find the location of the privacy policies and corresponding links on those websites.
3. The method for automatically extracting and restoring the structure of privacy policies for online services according to claim 1, characterized in that, The aforementioned plugin crawling method accesses the privacy policy webpage of the online service via a script, uses the SingleFile plugin to crawl the original HTML file of the privacy policy webpage, and obtains the HTML document of the online service's privacy policy.
4. A system for automatically extracting and restoring the structure of privacy policies for online services as described in any one of claims 1 to 3, characterized in that, include: The system comprises three modules: a privacy policy crawling module, a privacy policy subject identification module, and a privacy policy structure reconstruction module. Specifically: the privacy policy crawling module locates privacy policies on online service websites using keywords and obtains the original HTML file of the privacy policy webpage using the SingleFile plugin; the privacy policy subject identification module determines the location of the privacy policy subject in the HTML file based on the text distribution patterns of privacy policy nodes and extracts all nodes of the privacy policy subject in the HTML file; and the privacy policy structure reconstruction module extracts features such as the style of the first sequence number tag, text length, parent node tag name, font size, font weight, node depth, whether italicized, and whether it contains underscores, and uses machine learning to build a classifier to identify whether each node is a title and the title's hierarchy.
5. The system according to claim 4, characterized in that, The privacy policy crawling module includes a privacy policy locating unit and a privacy policy crawling unit. The privacy policy locating unit locates the privacy policy of an online service by searching for privacy policies and privacy terms on the online service website, and then jumps to the privacy policy webpage of the online service. The privacy policy crawling unit uses the SingleFile plugin to crawl the original HTML file of the privacy policy webpage to obtain the HTML document of the privacy policy of the online service.
6. The system according to claim 4, characterized in that, The privacy policy body identification module recursively finds the node containing the body of the privacy policy in the privacy policy HTML document by extracting the body text based on the uniformity of node text distribution. That is, it finds the smallest parent node containing all the body content in the HTML DOM tree.
7. The system according to claim 4, characterized in that, The privacy policy structure restoration module extracts features such as the style of the first serial number tag, text length, parent node tag name, font size, font weight, node depth, whether it is italic, and whether it has an underline. It then uses machine learning to build a classifier to identify whether each node is a title and the title level.