System and method for website data collection strategy analysis based on large language models

The website data collection strategy analysis system based on a large language model solves the problem of lengthy and inconsistent formatting of website data collection strategy documents, and achieves efficient and accurate data collection strategy analysis, improving the accuracy and efficiency of automated analysis.

CN120611080BActive Publication Date: 2025-11-04ZHEJIANG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511121332.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-08-12
Publication Date
2025-11-04
Estimated Expiration
2045-08-12

AI Technical Summary

Technical Problem

Existing technologies struggle to perform comprehensive and accurate automated analysis of website data collection strategies, especially due to the lengthy and inconsistent formats of data collection strategy documents, which results in insufficient automated analysis capabilities.

Method used

The system employs a website data collection strategy analysis module based on a large language model, which includes an input module, a data collection strategy crawling module, a data collection strategy preprocessing module, a website data collection clause coverage analysis module based on a large language model, and a personal data collection practice analysis module. It performs efficient and accurate analysis of website data collection strategies through a large language model.

Benefits of technology

It has achieved a complete automated process from target website domain name to analysis report, reducing manual intervention, improving processing efficiency, and ensuring intelligent parsing and rapid processing of complex documents, thereby improving the accuracy and consistency of analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120611080B_ABST
    Figure CN120611080B_ABST
Patent Text Reader

Abstract

The application provides a website data collection strategy analysis system and method based on a large language model, and belongs to the technical field of website data collection strategy analysis.The application reads a target website domain name and a predefined data collection clause, identifies and crawls relevant page content as a candidate page to extract a high-confidence data collection strategy page;then, a website data collection clause coverage analysis module based on a large language model takes the segmented page fragments as input to analyze the data collection clauses followed and not followed by the target website; a website personal data collection practice analysis module based on a large language model takes the same fragments as input to analyze the personal information types, collection intentions and third-party service types;finally, the clause coverage and personal data practice analysis results are integrated to generate a final data collection strategy analysis report.The application has perfect understanding and analysis capabilities, reasoning capabilities, and can realize accurate analysis and induction of website data collection strategy practice.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of website data collection strategy analysis, and in particular to a website data collection strategy analysis system and method based on a large language model. BACKGROUND

[0002] Under the background of the rapid development of informationization and digitization, strengthening personal information protection and standardizing the acquisition and use of personal information are not only related to the maintenance of personal rights and interests, but also related to the healthy development of the digital economy. Therefore, most online service websites are required to disclose their data collection practices through data collection strategy documents, such as exploring how online services collect user information and the rights that users can exercise with respect to their personal data. By analyzing website data collection strategies, users can be provided with a clear and intuitive description of website data collection practices, thereby enhancing citizens' sense of digital security and promoting the healthy development of the digital economy.

[0003] However, considering that data collection strategy documents are usually lengthy, complex and have different formats, there are still challenges in automated data collection strategy analysis. Existing technologies generally use unsupervised and rule-based methods or methods of training fine-tuned machine learning classifiers, which are limited to analyzing a small number of pre-defined data collection clauses. Therefore, there is still a lack of comprehensive understanding and analysis of website data collection practices.

[0004] Therefore, there is an urgent need for a precise and reliable automated website data collection strategy analysis method and system, which is of great significance to data security. SUMMARY

[0005] In view of the above background, the present application aims to provide a website data collection strategy analysis system and method based on a large language model, which has strong understanding and analysis capabilities and reasoning capabilities, and can efficiently and accurately analyze website data collection strategies.

[0006] To achieve the above-mentioned application purposes, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a website data collection strategy analysis system based on a large language model, comprising:

[0008] an input module for reading a target website domain name and pre-defined website data collection clauses to be analyzed;

[0009] a data collection strategy crawling module for identifying and crawling page content related to data collection strategies from the target website as candidate pages;

[0010] a data collection strategy preprocessing module for filtering candidate pages and extracting high-confidence data collection strategy pages;

[0011] The website data collection clause coverage analysis module based on a large language model takes the context fragments segmented from the high-confidence data collection policy page as input, and uses a large language model to analyze the data collection clauses that the target website complies with and does not comply with;

[0012] The website personal data collection practice analysis module based on a large language model takes the context fragments segmented from the high-confidence data collection policy page as input, and uses a large language model to analyze the types of personal information collected by the target website, the intention of collecting personal information, and the types of third-party services that receive personal information.

[0013] The output module is configured to integrate the analysis results of the website data collection clause coverage analysis module based on a large language model and the website personal data collection practice analysis module based on a large language model, and comprehensively output a data collection strategy analysis report of the target website.

[0014] As a preferred embodiment of the present application, the predefined website data collection clauses to be analyzed include the types of personal information collected, the types of personal information disclosed, the types of personal information sources, the intention of collecting personal information, the intention of disclosing personal information, the types of third-party services that receive personal information, and the retention period of personal information.

[0015] As a preferred embodiment of the present application, the data collection strategy crawling module is specifically configured as:

[0016] Extract keywords related to data collection clauses from a pre-set multilingual word list, and identify hyperlinks in the target website homepage that potentially point to data collection policy pages through keyword matching technology, and use the pages pointed to by the hyperlinks as candidate data collection policy pages.

[0017] Load the candidate data collection policy pages and download the HTML format content, remove the template file content and convert it into a Markdown format document. Each candidate data collection policy page corresponds to a document.

[0018] As a preferred embodiment of the present application, the data collection strategy preprocessing module is specifically configured as:

[0019] Apply a pre-trained machine learning classifier to preliminarily screen the candidate pages.

[0020] Filter the results of the preliminary screening through the URL keyword, and retain the candidate pages that meet the URL keyword. If none of the preliminary screening results meet the requirements, skip the URL keyword filtering process.

[0021] Match the target website domain name or redirection domain name based on the second-level domain name, exclude irrelevant pages, and use the remaining results as high-confidence data collection policy pages.

[0022] The document corresponding to the high-confidence data collection strategy page is segmented into segments that meet the context window limit of the large language model.

[0023] As a preferred embodiment of the present application, the classification process of the pre-trained machine learning classifier is:

[0024] The document corresponding to the high-confidence data collection strategy page is input, and a feature vector is extracted based on the TextRank or PositionRank algorithm. The feature vector is input into a random forest model to calculate the probability that the document content meets the data collection strategy page. The page corresponding to the document content with a probability greater than the threshold is the preliminary screened candidate page.

[0025] As a preferred embodiment of the present application, the URL keyword uses data collection.

[0026] As a preferred embodiment of the present application, the website data collection clause coverage analysis module based on the large language model is specifically configured as:

[0027] The large language model is instructed to play the role of an expert and provide a task description, which refers to judging whether the input segment contains the pre-defined website data collection clause information to be analyzed, and outputting a JSON format result.

[0028] The union of the clause content contained in the output of the large language model is taken as the website data collection clause coverage evaluation result of the target website.

[0029] As a preferred embodiment of the present application, the website personal data collection practice analysis module based on the large language model is specifically configured as:

[0030] The large language model is instructed to play the role of an expert and provide a task description, which refers to classifying the sub-categories of personal data collection practices in the input segment, and outputting a JSON format result. The personal data collection practices include three categories: the type of collected personal information, the intention of collecting personal information, and the type of third-party services receiving personal information. Each category of personal data collection practice contains several sub-categories.

[0031] The union of each sub-category output by the large language model is taken as the website personal data collection practice analysis evaluation result of the target website.

[0032] In a second aspect, the present application provides a website data collection strategy analysis method based on the above-mentioned website data collection strategy analysis system, which includes the following steps:

[0033] (1) Defining the website data collection clause to be analyzed;

[0034] (2) input the target website domain name;

[0035] (3) identify and crawl the page content related to the data collection strategy from the target website as a candidate page;

[0036] (4) used for filtering operation on the candidate page, extracting high confidence data collection strategy page;

[0037] (5) using a large language model to analyze the target website's compliance and non-compliance with data collection terms with the context segment cut from the high confidence data collection strategy page as input;

[0038] (6) using a large language model to analyze the types of personal information collected by the target website, the intention of collecting personal information, and the types of third-party services receiving personal information with the context segment cut from the high confidence data collection strategy page as input;

[0039] (7) integrating the analysis results of the website data collection terms coverage analysis module based on the large language model and the website personal data collection practice analysis module based on the large language model, and comprehensively outputting the data collection strategy analysis report of the target website.

[0040] In a third aspect, the present application provides a computer readable storage medium, the storage medium stores a computer program, when the computer program is executed by a processor, the website data collection strategy analysis method described above is realized.

[0041] In a fourth aspect, the present application provides a computer electronic device, comprising a memory and a processor;

[0042] The memory is used to store a computer program;

[0043] The processor is used to realize the website data collection strategy analysis method described above when the computer program is executed.

[0044] Compared with the prior art, the beneficial effects of the present application are:

[0045] (1) The present application does not depend on fixed analysis rules, realizes the complete automatic process from inputting the target website domain name to outputting the analysis report, including automatically crawling page content, filtering high confidence strategy page, and analysis module based on large language model. Compared with the unsupervised rule method or fine-tuning classifier in the prior art, the present application does not depend on artificial pre-defined rules or fixed analysis framework, significantly reduces the artificial intervention, and improves the processing efficiency.

[0046] (2) This invention fully utilizes the natural language processing advantages of the large language model to achieve intelligent parsing of complex documents. In the analysis module based on the large language model, the large language model is instructed to play the role of an expert, performing analysis on the coverage of website data collection terms and analysis on personal data collection practices, thus solving the technical problem of lengthy and complex data collection strategy documents. Considering the contextual limitations of the large language model, page segmentation technology is adopted to ensure the rapid processing of large-scale documents, improving the overall analysis efficiency while maintaining the continuity of the end-to-end process. Attached Figure Description

[0047] Figure 1 This is a general design diagram of a website data collection strategy analysis system based on a large language model.

[0048] Figure 2 A flowchart illustrating the website data collection strategy analysis method based on a large language model. Detailed Implementation

[0049] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0050] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0051] This invention provides a website data collection strategy analysis system based on a large language model. By predefining the data collection terms of the website to be analyzed, it ultimately generates analysis results on whether the target website meets the requirements of each predefined term.

[0052] like Figure 1 As shown, the website data collection strategy analysis system mainly includes an input module, a data collection strategy crawling module, a data collection strategy preprocessing module, a website data collection clause coverage analysis module based on a large language model, a website personal data collection practice analysis module based on a large language model, and an output module.

[0053] The functional configurations of each module are described below:

[0054] (1) Input module

[0055] The input module is used to read the target website domain name and predefined data collection terms for the website to be analyzed.

[0056] The data collection policy contents that the predefined website needs to declare include: the type of collected personal information, the type of disclosed personal information, the type of personal information source, the intention of collecting personal information, the intention of disclosing personal information, the type of third-party service receiving personal information, and the retention period of personal information. Generally, the type of personal information source includes user-provided information, advertising networks, and data intermediaries; the type of personal information includes financial data (credit card data), health conditions, biometric data (fingerprint), contact information, geographic location, social media data, IP address, computer model, user online activity, etc.; the intention of collecting or disclosing personal information includes providing services (storing your account password for subsequent login verification), advertising, analysis and research (better understanding of user preferences), customized services, legal requirements, etc.; the type of third-party service receiving personal information includes specifically named third-party services (such as Google Analytics), classified third-party services (such as advertising networks), public / other users (such as microblog users who can see your information), etc.

[0057] It should be noted that the type of collected personal information and the type of disclosed personal information overlap but are not completely the same. For example, for target website A, the user's biometric data (fingerprint) will be obtained as a login password, which belongs to the type of collected personal information, but will not be disclosed, i.e., does not belong to the type of disclosed personal information.

[0058] The personal information processing activities involved in the implementation of the present application, including but not limited to sensitive data such as financial data and biometric data, strictly follow the requirements of current Chinese laws, regulations and industry standards.

[0059] The above data collection clauses can also be designed according to the needs of those skilled in the art.

[0060] (2) Data collection policy crawling module

[0061] The data collection policy crawling module is used to identify and crawl the page content related to the data collection policy from the target website as a candidate page.

[0062] It should be noted that a website is a collection of a set of interrelated website pages, media files, and other digital resources (such as images, stylesheets, and script files) that are connected to each other through hyperlinks and have a common root domain name. In the target website of the present application, at least one page contains content related to the data collection policy.

[0063] Specifically, the implementation process of the data collection policy crawling module includes the following sub-steps:

[0064] (2.1) Extract keywords related to data collection from existing multilingual glossaries, and identify potential hyperlinks in the target website homepage that point to data collection policy pages through keyword matching techniques. The pages pointed to by the hyperlinks are considered as candidate data collection policy pages.

[0065] (2.2) For the target website homepage and its candidate data collection policy pages, use web data collection tools such as Selenium to load the pages, and download the page content in HTML format after complete loading.

[0066] (2.3) Remove template file content such as script code, style sheets, and tags (header, footer, nav, etc.) from the page content, and the remaining content is considered as core text content. Further convert the text content into Markdown format documents that are easy for large language models to understand through conversion tools such as html-to-markdown, and perform subsequent processing on a page-by-page basis.

[0067] (3) Data collection strategy preprocessing module

[0068] The data collection strategy preprocessing module is used to filter the candidate pages and extract high-confidence data collection strategy pages.

[0069] Specifically, the filtering operation on the candidate data collection strategy pages refers to: first, for a target website that has at least one candidate data collection strategy page, perform preliminary screening based on a pre-trained data collection strategy content machine learning classifier; in this embodiment, the Markdown format document corresponding to the candidate data collection strategy page is used as input, and algorithms such as TextRank and PositionRank are used to extract feature vectors, and a random forest model is used to determine the probability of the page belonging to a data collection strategy page, with a threshold of 0.5. Pages with a probability higher than 0.5 are retained, and the rest are discarded, resulting in a set of pages after preliminary screening.

[0070] Secondly, considering that some terms of service or disclaimer pages may contain data collection strategy related content, leading to misclassification by the classifier, further candidate pages with "data collection" keywords in the URL of the website page are selected; here, if the set of pages after preliminary screening contains candidate pages with "data collection" keywords, only these candidate pages are retained for the next operation, if the set of pages after preliminary screening does not contain candidate pages with "data collection" keywords, this process is skipped and the set of pages after preliminary screening is directly subjected to the next operation.

[0071] Finally, considering that the target website may contain third-party service data collection policy descriptions that are irrelevant to its own data collection policy or follow the data collection policy of its parent company, the final high-confidence data collection policy page is determined as the page that meets any of the following criteria: (1) the candidate page whose second-level domain name matches the target website domain name; (2) the candidate page whose second-level domain name matches the target website redirect domain name.

[0072] The final several high-confidence data collection policy pages are segmented: considering the context window limit of the large language model, for the obtained high-confidence data collection policy pages, the Markdown format document of each page is segmented into segments that meet the length limit of the context window of the large language model, and the operation of the website data collection clause coverage analysis module based on the large language model and the website personal data collection practice analysis module based on the large language model is performed respectively.

[0073] This module uses a multi-stage filtering operation to first use a pre-trained machine learning classifier for preliminary screening, and then excludes irrelevant pages through URL keyword and second-level domain name matching to ensure the high reliability of the analysis results and effectively reduce false positives, improving analysis accuracy.

[0074] (4) Website data collection clause coverage analysis module based on large language model

[0075] The website data collection clause coverage analysis module based on the large language model takes the context segments segmented from the high-confidence data collection policy pages as input and uses the large language model to analyze the data collection clauses that the target website follows and does not follow.

[0076] Specifically, the website data collection clause coverage analysis aims to evaluate whether a given data collection policy segment covers the required content of a certain data collection clause.

[0077] The specific process of the website data collection clause coverage analysis based on the large language model includes: first, instructing the large language model to play the role of an expert in deep online data collection policy analysis; second, providing the task description to the large language model, which is in the format "Your task is to carefully review the provided data collection policy segment and determine whether it contains the information required by a certain data collection clause", where "a certain data collection clause" needs to be replaced with a preset data collection clause item, such as "explain the intention to collect consumer personal information"; third, specifying the basic format of the large language model's answer, which should be in JSON format and contain "whether the segment contains the corresponding data collection clause" and "the segment description corresponding to the data collection clause"; finally, providing the segmented target website data collection policy content segments to the large language model.

[0078] In this module, the large language model outputs the evaluation results of whether the given data collection strategy fragment covers the content required by a certain data collection clause, integrates all fragments of all high-confidence data collection strategy pages, takes the union of the covered content of the evaluation results as the evaluation results of the website data collection clause coverage of the target website.

[0079] (5) Website personal data collection practice analysis module based on large language model

[0080] The website personal data collection practice analysis module based on the large language model takes the context fragments segmented from the high-confidence data collection strategy pages as input, and uses the large language model to analyze the types of personal information collected by the target website, the intention of collecting personal information, and the types of third-party services receiving personal information.

[0081] Specifically, the website personal data collection practice analysis is committed to evaluating the practice of the given data collection strategy fragment on personal data information, including the types of personal information collected by the website, the intention of collecting personal information by the website, and the types of third-party services receiving personal information.

[0082] For each category, the specific process of the website personal data collection practice analysis based on the large language model includes: first, instruct the large language model to play the role of an expert in deep online data collection strategy analysis; second, provide the task description to the large language model, the task description format is "your task is to carefully review the provided data collection strategy fragment and classify a certain personal data collection practice involved in the fragment", where "a certain personal data collection practice" needs to be replaced with a preset personal data collection practice category, such as "types of personal information collected by the website"; third, provide all classification information subordinate to the current preset personal data collection practice category to the large language model, for example, the personal information category includes user device identifier, IP address, and session stub; then specify the basic format of the large language model's answer, which should be presented in JSON format and output "whether the fragment contains the corresponding subordinate classification of personal data collection practice" and "the fragment description corresponding to the corresponding subordinate classification" for each subordinate classification; finally, provide the segmented fragments of the target website data collection strategy content to the large language model.

[0083] In this module, the large language model outputs the evaluation results of whether the given data collection strategy fragment covers the content required by a certain data collection clause, integrates all fragments of all high-confidence data collection strategy pages, takes the union of the covered content of the evaluation results as the evaluation results of the website data collection clause coverage of the target website.

[0084] (6) Output module

[0085] The output module is used for integrating the analysis results of the website data collection clause coverage analysis module based on a large language model and the website personal data collection practice analysis module based on a large language model, and outputting a data collection strategy analysis report of the target website.

[0086] Embodiments of the system of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for running by the processor of the device with data processing capability.

[0087] Based on the same inventive concept, the embodiment also provides a website data collection strategy analysis method based on the above website data collection strategy analysis system, as shown in the method, specifically comprising the following steps: Figure 2

[0088] (1) defining the website data collection clauses to be analyzed;

[0089] (2) inputting the domain name of the target website;

[0090] (3) identifying and crawling the page content related to the data collection strategy from the target website as candidate pages;

[0091] (4) filtering the candidate pages to extract high-confidence data collection strategy pages;

[0092] (5) using a large language model to analyze the data collection clauses followed and not followed by the target website by taking the context segments segmented from the high-confidence data collection strategy pages as input;

[0093] (6) using a large language model to analyze the types of personal information collected by the target website, the intention of collecting personal information, and the types of third-party services receiving personal information by taking the context segments segmented from the high-confidence data collection strategy pages as input;

[0094] (7) integrating the analysis results of the website data collection clause coverage analysis module based on a large language model and the website personal data collection practice analysis module based on a large language model, and outputting a data collection strategy analysis report of the target website.

[0095] ​It should be noted that the website data collection strategy analysis method in the above embodiments can be essentially executed by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer electronic device corresponding to the method provided by the above embodiments, which includes a memory and a processor.

[0096] The memory is configured to store a computer program.

[0097] The processor is configured to implement the website data collection strategy analysis method in the above embodiments when executing the computer program.

[0098] In addition, the logical instructions in the above memory can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application.

[0099] Therefore, another preferred embodiment of the present application also provides a computer readable storage medium corresponding to the method provided by the above embodiments, and the storage medium stores a computer program. When the computer program is executed by a processor, the website data collection strategy analysis method in the above embodiments can be implemented.

[0100] It can be understood that the above storage medium can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0101] It can be understood that the processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0102] In addition, it should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method is only a logical functional division, and there can be another division mode in actual implementation, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0103] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A website data collection strategy analysis system based on a large language model, characterized in that, include: The input module is used to read the target website domain name and predefined data collection terms for the website to be analyzed; The data collection strategy crawling module is used to identify and crawl page content related to the data collection strategy from the target website as candidate pages; The data collection strategy preprocessing module is used to filter candidate pages and extract high-confidence data collection strategy pages. The website data collection terms coverage analysis module based on the large language model takes context fragments segmented from high-confidence data collection strategy pages as input and uses the large language model to analyze the data collection terms that the target website complies with and does not comply with. The "Website Personal Data Collection Practice Analysis Module Based on Large Language Model" takes context fragments segmented from high-confidence data collection strategy pages as input and uses large language model to analyze the types of personal information collected by the target website, the intent to collect personal information, and the types of third-party services that receive personal information. The output module integrates the analysis results from the website data collection clause coverage analysis module based on the large language model and the website personal data collection practice analysis module based on the large language model, and outputs a comprehensive data collection strategy analysis report for the target website.

2. The website data collection strategy analysis system based on a large language model according to claim 1, characterized in that, The predefined data collection terms of the websites to be analyzed include the types of personal information collected, the types of personal information disclosed, the types of sources of personal information, the intent to collect personal information, the intent to disclose personal information, the types of third-party services that receive personal information, and the retention period of personal information.

3. The website data collection strategy analysis system based on a large language model according to claim 1, characterized in that, The specific configuration of the data collection strategy crawling module is as follows: Keywords related to data collection terms are extracted from a pre-defined multilingual vocabulary, and hyperlinks pointing to potential data collection strategy pages on the target website homepage are identified through keyword matching technology. The pages pointed to by these hyperlinks are then used as candidate data collection strategy pages. Load the candidate data collection strategy page and download the HTML content. After removing the boilerplate content, convert it to a Markdown document. Each candidate data collection strategy page corresponds to one document.

4. The website data collection strategy analysis system based on a large language model according to claim 3, characterized in that, The data collection strategy preprocessing module is specifically configured as follows: A pre-trained machine learning classifier is used to initially screen candidate pages; The results after initial screening are filtered by URL keywords, and candidate pages that meet the URL keywords are retained; if none of the results after initial screening meet the requirements, the URL keyword filtering process is skipped. Based on the second-level domain name, the target website domain name or redirected domain name is matched, irrelevant pages are excluded, and the remaining results are used as high-confidence data collection strategy pages. The document corresponding to the high-confidence data collection strategy page is segmented into fragments that meet the context window constraints of the large language model.

5. The website data collection strategy analysis system based on a large language model according to claim 4, characterized in that, The classification process of the pre-trained machine learning classifier is as follows: Using the documents corresponding to the high-confidence data collection strategy pages as input, feature vectors are extracted based on the TextRank or PositionRank algorithm. The feature vectors are then input into a random forest model to calculate the probability that the document content matches the data collection strategy pages. Pages corresponding to document content with a probability greater than a threshold are selected as candidate pages after preliminary screening.

6. The website data collection strategy analysis system based on a large language model according to claim 1, characterized in that, The specific configuration of the website data collection terms coverage analysis module based on the large language model is as follows: The large language model is instructed to act as an expert and provide a task description, which means determining whether the input fragment contains predefined data collection terms information of the website to be analyzed and outputting results in JSON format. Integrate all fragments from all high-confidence data collection strategy pages, and take the union of the included clause content output by the large language model to obtain the evaluation result of the target website's data collection clause coverage.

7. The website data collection strategy analysis system based on a large language model according to claim 1, characterized in that, The specific configuration of the website personal data collection practice analysis module based on the large language model is as follows: The large language model is instructed to act as an expert and provide a task description, which refers to classifying the subcategories of personal data collection practices in the input fragment and outputting results in JSON format. The personal data collection practices include three categories: the types of personal information collected, the intent to collect personal information, and the types of third-party services that receive personal information. Each category of personal data collection practices contains several subcategories. All fragments from all high-confidence data collection strategy pages are integrated, and the union of the various subcategories output by the large language model is used as the analysis and evaluation results of the target website's personal data collection practices.

8. A method for analyzing website data collection strategies based on the website data collection strategy analysis system of claim 1, characterized in that, Includes the following steps: (1) Define the data collection terms for the website to be analyzed; (2) Enter the target website domain name; (3) Identify and crawl page content related to the data collection strategy from the target website as candidate pages; (4) Used to filter candidate pages and extract high-confidence data collection strategy pages; (5) Using context fragments segmented from high-confidence data collection strategy pages as input, large language models are used to analyze the data collection terms that the target website complies with and does not comply with; (6) Using context fragments segmented from the high-confidence data collection strategy page as input, a large language model is used to analyze the types of personal information collected by the target website, the intent to collect personal information, and the types of third-party services that receive personal information; (7) Integrate the analysis results of the website data collection clause coverage analysis module based on the big language model and the website personal data collection practice analysis module based on the big language model, and output a comprehensive data collection strategy analysis report for the target website.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the website data collection strategy analysis method as described in claim 8.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the website data collection strategy analysis method as described in claim 8 when executing the computer program.

Citation Information

Patent Citations

  • APP private data use purpose consistency analysis method based on large language model

    CN117896732A

  • Method and system for obtaining website API (Application Program Interface) by one key

    CN118838590A