A method and system for collecting breadth data

Abstract

The present invention relates to the field of data collection, and in particular to a method and system for breadth data collection, including: establishing a web page meta information base; determining the daily scheduling unit download quota and the total download quota for the day; selecting a corresponding number of links from the web page meta information base and allocating the download quota; download process control; after post-processing and data cleaning of the downloaded text, entering the refill queue, and updating the web page meta information base through information refill. The beneficial effects of the present invention are: discovering as many web page links as possible and reducing the flow impact on the website; analyzing the downloaded content, inferring the quality of the undownloaded connections, reducing the download of low-quality web pages and repeated downloads by allocating quotas through preferential download, improving data quality and download efficiency, and reducing the consumption of network resources during the data collection process; using a separate information refill queue to ensure the atomicity and stability of the modification operation of the web page meta information base.

Description

Technical Field

[0001] The present invention relates to the field of data collection, and in particular to a method and system for collecting breadth data. Background Art

[0002] In recent years, with the progress of artificial intelligence technology, the field of NLP natural language has made great progress. Many large language models (LLMs) have been trained and applied in the field of natural language processing to study various theories and methods for achieving effective communication between people and computers using natural language. The training of large language models requires the construction of a high-quality and diverse large language model dataset, which requires the collection and processing of web page data to obtain a large amount of high-quality text information as the input of the model for training large language models. However, there are many problems with existing data collection technologies, such as the inability to obtain complete links when collecting complex sites; it is easy to over-download, causing the other party's website to crash; the download page does not perform content quality analysis and inference, resulting in repeated downloads or low-quality downloads, affecting the efficiency of data collection.

[0003] Therefore, in the process of acquiring large amounts of web page data, it becomes crucial to collect Internet data quickly, accurately, securely and efficiently. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings, the present invention aims to provide a method and system for collecting breadth data quickly, accurately, safely and efficiently, so as to selectively download high-quality web pages and improve download efficiency.

[0005] The present invention achieves the above object through the following scheme: A method for collecting breadth data, comprising the following steps:

[0006] S1: Establishing web page meta information database;

[0007] S2: Determine the download task parameters for the day, including the download quota for each scheduling unit and the total download quota for the day;

[0008] S3: Select a corresponding number of links from the web page meta-information library and allocate download quotas;

[0009] S4: download process control;

[0010] S5: Download the text for post-processing and data cleaning, and the processed text enters the re-injection queue;

[0011] S6: Update the web page meta-information library through information re-injection.

[0012] Preferably, the web page meta information library in S1 uses a standardized URL as a key to store URL meta information; the URL includes downloaded URLs and non-downloaded URLs.

[0013] Preferably, the scheduling unit in S2 is one or a group of hosts, and the scheduling unit download quota refers to setting a crawling upper limit for each scheduling unit every day; the total download quota is the total number of URLs that can be downloaded and processed in one day.

[0014] Preferably, the specific steps of S3 include:

[0015] S301: Generate a first pattern set according to the downloaded link processing;

[0016] S302: Filter the first pattern set to obtain a filtered first pattern set set;

[0017] S303: Process the undownloaded links to generate a second pattern set;

[0018] S304: Obtain the intersection of the second pattern set and the filtered first pattern set, and implement the attribution of the undownloaded links to obtain a third pattern set;

[0019] S305: Determine the scheduling unit corresponding to each third mode set, and allocate a download quota to each scheduling unit;

[0020] S306: Select a download link in each scheduling unit.

[0021] Preferably, the specific implementation of step S301 is to divide the downloaded links in the web page meta information library into two parts: prefix and suffix, then enumerate the sets of all inclusion relationships between the prefix and suffix, and calculate the Cartesian product of the two sets to obtain the first pattern set.

[0022] Preferably, in step S302, a download link number threshold is preset, pattern sets with download link numbers greater than the threshold and pattern sets at the top level of each domain are retained, and the remaining pattern sets are discarded to obtain a filtered first pattern set set.

[0023] Preferably, in step S303, the undownloaded links in the web page meta information library are divided into two parts: a prefix and a suffix. Then, all sets of inclusion relationships between the prefix and the suffix are enumerated, and the Cartesian product of the two sets is calculated to obtain a second pattern set.

[0024] Preferably, in step S303, the elements in the intersection are sorted from high to low according to a priority rule, and the link is assigned to a pattern set with a higher priority; the priority is defined as: number of suffix-determined items > number of prefix-determined items > number of suffix-indetermined items > number of prefix-indetermined items > validity.

[0025] Preferably, the specific steps of S4 include:

[0026] S401: Obtaining a website access pressure curve;

[0027] S402: According to the pressure curve of the dispatching unit, adjustment is performed according to the formula:

[0028]

[0029] Where n is the total daily download volume, a is the control coefficient, which usually ranges from 1 to 2, and T i Delay for sharding time;

[0030] S403: The downloader assigns tasks for downloading.

[0031] Preferably, the re-injection queue refers to a pre-deployed queue service that uses a first-in-first-out method to implement updates to the web page meta-information library; all information that has undergone post-processing and data cleaning needs to be pushed to the re-injection queue before entering the update of the web page meta-information library, and then the web page meta-information library is updated by a single information re-injection program.

[0032] A system for collecting breadth data includes: a web page meta information base, an aggregation module, a quota allocation module, a link selection module, a downloader, a data post-processing and data cleaning module, and a re-injection queue module; URL meta information in the web page meta information base is aggregated twice by the aggregation module to obtain a third pattern set, the quota allocation module determines the scheduling unit corresponding to each third pattern set, and allocates the daily download quota of the scheduling unit to each scheduling unit; the link selection module selects the link of the corresponding quota from the undownloaded link set corresponding to the third pattern set according to the daily download quota, and the downloader completes the download task, performs preliminary diversion analysis and refined data cleaning on the downloaded text content, and uses multiple quality models of different dimensions to score, pushes the URL of the web page and its corresponding quality score to the re-injection queue, and realizes the update of the web page meta information base.

[0033] The beneficial effects of the present invention are: 1. Find as many web links as possible, and control the number of visits to each website during the download process to reduce the traffic impact on the website; 2. Analyze the downloaded content, infer the quality of the undownloaded connections, and reduce the download of low-quality web pages and repeated downloads by allocating quotas through preferential download, improve data quality and download efficiency, and reduce the consumption of network resources during data collection; 3. Use a separate information re-injection queue to ensure the atomicity and stability of the modification operation of the web page meta-information library. Through the present invention, more high-quality web page text information can be obtained as much as possible under limited resources, and the impact on the website can be reduced, so as to achieve fast, accurate, safe and efficient collection of Internet data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the steps of the method of the present invention;

[0035] Figure 2 is a schematic flow chart of step S3 of the method of the present invention;

[0036] Figure 3 is a schematic flow chart of step S4 of the method of the present invention;

[0037] Figure 4 It is a schematic diagram of the structure of the system of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with specific implementation examples, but the protection scope of the present invention is not limited thereto:

[0039] Example: Figure 1 is a flow chart of an embodiment disclosed in the present invention, which provides a method for collecting breadth data, specifically comprising:

[0040] S1: Establish a web page meta information database.

[0041] URL (Uniform Resource Locator) is the address of a resource on the Internet. URL consists of a protocol (scheme), a host name (host), a path (path), and a query string (query), such as "http: / / www.example.com / page.html". URL is used to specify the location to access a specific resource on the Internet. In the URL, the host part usually indicates the address of the server to be accessed, which is the place where services are provided or data is stored, and is located in the first half of the URL. Domain is the primary domain name where the URL host is located, such as "example.com".

[0042] The web page meta information library (linkbase) is the core database of this system. In a specific embodiment, the web page meta information library uses a standardized URL as a key to store URL meta information related to the implementation of this method, including URLs that have been downloaded and URLs that have not been downloaded. The URL meta information includes: discovery information. The discovery information uses simple text to describe when and why the URL was discovered, as well as the best parent link (source). For URLs that have been downloaded, it also includes download information and feedback information. The download information includes the download time, the corresponding HTTP status code, and the storage path of the downloaded web page. The feedback information includes information such as the language of the web page content, page type, quality score, and repetition.

[0043] The web page meta information library may adopt a distributed Key-Value database that meets high performance requirements, such as a RocksDB cluster, HBase, etc.

[0044] S2: Determine the download task parameters for the day, including the download quota for each scheduling unit and the total download quota for the day.

[0045] In the specific implementation, it is necessary to determine the download quota of each scheduling unit and the total download quota of the day. The total download quota of the day is the total number of URLs that can be downloaded and processed in one day. The basis for determining the total download quota of the day is subject to the resource limitations of the downloader and post-text processing, including the computing, storage, bandwidth and other resources of the downloader and post-processing. The download quota of the scheduling unit refers to setting a crawling limit for each scheduling unit every day in order to prevent excessive downloads from causing an impact on the website. The source of the download quota of the scheduling unit can be calculated through online tools for website traffic and market analysis, such as similarweb, which provides monthly host visits, Google index volume, etc.

[0046] The scheduling unit is usually one or a group of hosts, such as *.ycombinator.com, v.qq.com, where * represents any match. Generally speaking, for small and medium-sized sites, one domain is a scheduling unit, corresponding to all hosts in the domain; for large sites, one host is a scheduling unit.

[0047] S3: Select a corresponding number of links from the web page meta information library and allocate download quotas.

[0048] According to the total download quota of the day, select the number of undownloaded links from the web page meta information library for download on the day. In order to reduce the impact on the crawled website, it is necessary to ensure that the number of links selected by each scheduling unit does not exceed the download quota of the scheduling unit in the process of allocating download quotas. At the same time, in order to use limited resources to obtain more high-quality web page text data information, it is necessary to infer the quality of undownloaded links based on the quality of downloaded links, and give priority to high-quality links.

[0049] like Figure 2 As shown, the specific implementation method of this step is as follows:

[0050] S301: Generate a first pattern set according to the downloaded link processing.

[0051] The pattern set refers to a set of links clustered according to certain rules in a scheduling unit. In a specific implementation, it is necessary to cluster the links in the scheduling unit, and in the same clustered pattern set, the quality of the pattern set is scored by the score of the downloaded link to achieve an overall judgment of the link quality of the clustered pattern set.

[0052] In a specific embodiment, the implementation method is:

[0053] First, split the downloaded links in the web page meta-information library into two parts: prefix and suffix. Then, enumerate the sets of all inclusion relationships between the prefix and the suffix, and calculate the Cartesian product of the two sets to obtain the first pattern set. The segmentation method is to split according to the host and path parts of the URL. The host part is the prefix, the path part is the suffix, and the query part is not considered. The inclusion relationship, for the prefix, refers to the prefix itself and *.domain, where * represents a wildcard. For the suffix, it refers to each item of the suffix subpath, which can be a wildcard or an upper-level path, and ends with / . Specifically, for the link abcd / p1 / p2 / p3, where cd is the domain, the prefix is ​​abcd, and the inclusion relationship set is abcd and *.cd, with a total of 2 elements. The suffix is ​​ / p1 / p2 / p3, and the inclusion relationship set is / p1 / p2 / p3, / p1 / p2 / *,

[0054] / p1 / * / p3, / * / p2 / p3, / p1 / *, / * / p2 / *, / * / p3, / *, there are 8 elements in total. The Cartesian product of the two sets gives a total of 16 pattern sets, including: abcd / p1 / p2 / p3, abcd / p1 / p2 / *, abcd / p1 / * / p3, abcd / * / p2 / p3, abcd / p1 / *, abcd / * / p2 / *, abcd / * / p3, abcd / *,

[0055] *.cd / p1 / p2 / p3, *.cd / p1 / p2 / *, *.cd / p1 / * / p3, *.cd / * / p2 / p3, *.cd / p1 / *, *.cd / * / p2 / *, *.cd / * / p3, *.cd / *. *.cd / * can include all links of the domain and is defined as the top-level pattern set of the domain.

[0056] It is worth noting that there may be intersections between different first pattern sets, and there is a many-to-many relationship between links and first pattern sets, that is, a link may belong to multiple first pattern sets, and a first pattern set may include multiple links.

[0057] The above correspondence is aggregated for the first time according to the pattern set, so as to obtain the links and related attributes corresponding to each pattern set. The aggregation method is to merge the links with the same first pattern set under the first pattern set. For example, after the Cartesian product of Link 1, the pattern sets a and b are obtained, and after the Cartesian product of Link 2, the pattern sets a and c are obtained. After aggregation, the link set of pattern set a is (Link 1, Link 2).

[0058] The related attributes include the number of links of the pattern set, the link quality score and the validity of the pattern set. The link quality score is the score after the downloaded link text content is cleaned (extracted, deduplicated, and scored), and the higher the score, the better the quality, and failure to pass the cleaning is recorded as 0 points; the validity of the pattern set refers to the download revenue of the pattern set, and the calculation formula is: the sum of the quality scores of the downloaded links in the pattern set / the number of downloaded links.

[0059] S302: Filter the first pattern set to obtain a filtered first pattern set set.

[0060] In a specific embodiment, the first pattern set is filtered, and the pattern sets with fewer download links are unified into the top-level pattern set. Specifically, a download link number threshold is preset, and the pattern sets with download links greater than the threshold and the pattern sets at the top level of each domain are retained, and the remaining pattern sets are discarded to obtain the filtered pattern sets. After the above operation is completed, the filtered set of the first pattern sets is obtained.

[0061] S303: Process the undownloaded links to generate a second pattern set.

[0062] The non-downloaded links in the web page meta information library are divided into two parts: prefix and suffix, and then the sets of all inclusion relationships of the prefix and suffix are enumerated, and the Cartesian product of the two sets is calculated to obtain the second pattern set; specifically, for the non-downloaded link bcd / p1 / p4, its prefix is ​​bcd, domain is *.cd, the inclusion relationship set is bcd, *.cd, the suffix is ​​ / p1 / p4, and the inclusion relationship set is / p1 / p4, / p1 / *, / * / p4, / *. After calculating the Cartesian product, a total of 8 pattern sets are obtained.

[0063] S304: Obtain the intersection of the second pattern set and the filtered first pattern set, and implement the attribution of the undownloaded links to obtain a third pattern set.

[0064] Take the intersection of the second pattern set of undownloaded links and the filtered first pattern set. The elements in the intersection are sorted from high to low according to the priority rule, and the links are assigned to the pattern set with higher priority to obtain the third pattern set. The priority is defined as "number of suffix-determined items > number of prefix-determined items > number of suffix-indetermined items > number of prefix-indetermined items > validity". Among them, the prefix-determined item refers to the number of non-wildcards after the prefix is ​​split according to '.', the prefix-indetermined item refers to the number of wildcards after the prefix is ​​split according to '.', the suffix-determined item refers to the number of non-wildcards after the suffix is ​​split according to ' / ', and the suffix-indetermined item refers to the number of wildcards after the prefix is ​​split according to ' / '.

[0065] According to the above priority rules, in the above example of S301, assuming that the 16 pattern sets are the final pattern sets after the first aggregation and filtering, the corresponding priority order is:

[0066] Prefix priority abcd>*.cd;

[0067] Suffix priority / p1 / p2 / p3> / p1 / p2 / *= / p1 / * / p3= / * / p2 / p3> / p1 / *= / * / p2 / *= / * / p3> / *;

[0068] The intersection of the second pattern set and the filtered first pattern set is the prefix *.cd, the suffix / p1 / *, / *, and the Cartesian product is *.cd / p1 / *, *.cd / *. According to the priority rule, *.cd / p1 / *>*.cd / *, so the link belongs to the *.cd / p1 / * pattern set.

[0069] After determining the attribution relationship, the download links are aggregated for the second time according to the third pattern set to obtain a set of undownloaded links corresponding to each third pattern set, where the number of undownloaded links is the number of schedulable links. It is worth noting that at this time, there is a many-to-one relationship between the undownloaded link set and the third pattern set.

[0070] S305: Determine the scheduling unit corresponding to each third mode set, and allocate a download quota to each scheduling unit.

[0071] Determine the scheduling unit corresponding to each third pattern set, and allocate the daily download quota of the scheduling unit to each scheduling unit. In a specific embodiment, it is first necessary to confirm the corresponding scheduling unit of each third pattern set, and the confirmation method is that if the prefix part of the third pattern set does not contain a wildcard and the prefix is ​​included in the scheduling unit set, then the scheduling unit is the prefix of the pattern set, otherwise the scheduling unit is the domain corresponding to the prefix. Accordingly, in the aforementioned example, the scheduling unit for the pattern set abcd / * is abcd, and the scheduling unit for the pattern set *.cd / * is cd.

[0072] Then, all mode sets are sorted in reverse order according to effectiveness in the scheduling unit, and the download quota of each scheduling unit is allocated to the number of schedulable links of the corresponding mode set according to effectiveness from high to low, so as to obtain the actual number of downloadable links of the mode set.

[0073] Finally, all the mode sets assigned to the download quota of the scheduling unit are globally sorted in reverse order according to their effectiveness, and the total download quota of the day is allocated to the actual number of downloadable links of the corresponding mode set from high to low according to the effectiveness, so as to obtain the download quota of the day allocated to each mode set.

[0074] S306: Select a download link in each scheduling unit.

[0075] According to the daily download quota allocated to each mode set, a link of a corresponding quota is selected from the set of undownloaded links corresponding to the third mode set as the final selection result.

[0076] The selection process may be purely random, or certain selection rules may be set according to information such as link depth, link discovery time, etc.

[0077] S4: Download process control: The links selected every day are divided according to the scheduling units they belong to. The links that need to be downloaded by each scheduling unit are distributed to the download within one day according to the access pressure curve of the scheduling unit. Figure 3 The specific operation method is as follows:

[0078] S401: Obtaining the access pressure curve of the website.

[0079] The website access pressure curve is derived based on the website request data, where the request data may be provided by a third-party data provider.

[0080] The website's stress value is obtained by comparing the actual number of requests and the maximum number of requests during a certain period of time, thereby obtaining a stress curve. For example, if a website can withstand 1000 requests per minute and its number of Internet requests is only 500 per minute during a certain period of time, then the stress value for that period of time is (0.5, 1000). Usually, in order to minimize the impact of data crawling on external websites, crawling will be done when the other party's website is relatively idle. For example, domestic websites are usually relatively idle in the early morning, while American websites are the opposite due to time zone reasons.

[0081] S402: Adjust the pressure curve.

[0082] According to the pressure curve of the dispatching unit, adjust according to the following formula

[0083]

[0084] Where n is the total daily download volume, a is the control coefficient, which usually ranges from 1 to 2, and T i The delay for the fragmentation time.

[0085] S403: The downloader assigns tasks for downloading.

[0086] After the downloader assigns the task, it downloads. In a specific embodiment, the webpage content can be downloaded by means of http1.1 protocol request, and the downloader can use a package based on libcurl. The downloaded webpage content is pushed to the queue for temporary storage through the self-developed message queue.

[0087] S5: Download the text for post-processing and data cleaning, and the processed text enters the re-injection queue.

[0088] The downloaded webpage needs to be further processed for text content before it can be used as data for model training. The processing steps include post-processing and data cleaning. The processed text enters the re-injection queue. The re-injection queue refers to a pre-deployed queue service that uses first-in-first-out to implement the update of the webpage meta-information library.

[0089] First, post-processing is performed to perform preliminary analysis and diversion of web page content. The post-processing operation mainly includes:

[0090] Web page content analysis: Language identification, using language models to identify the language of web page text and determine the language type; spam detection, using corresponding models and algorithms to detect web pages and other content, and additionally mark such content for subsequent crawling suppression and link deletion.

[0091] Link promotion: By performing a depth-first traversal of the web page, the links it references are selected and normalized, that is, the web page URL is standardized and pushed into the re-injection queue as the web page target for subsequent potential downloads.

[0092] Classified storage: For downloaded text content, zstd compression is used to store it in different files according to different languages.

[0093] For the post-processed web page text content, further data cleaning is required.

[0094] The steps include:

[0095] 1. Extract the core body text in HTML;

[0096] 2. Based on minhash, global deduplication of text is performed;

[0097] 3. Perform fine-grained text cleaning: including removing useless content by line, normalizing some descriptions, etc.;

[0098] 4. For the refined cleaned text, perform vector deduplication on the text;

[0099] 5. For the final deduplicated text, multiple quality models of different dimensions are used to score and obtain the webpage text quality score, and the URL of the webpage and its corresponding quality score are pushed to the re-injection queue. The quality model can be a BERT model. The one with the higher quality score is selected to enter the subsequent data processing flow to prepare for entering the large language model training corpus.

[0100] S6: Update the web page meta-information library through information re-injection.

[0101] All information that has been post-processed and cleaned needs to be pushed to the re-injection queue before entering the web page meta information library for update, and then a single information re-injection program will be used to update the web page meta information library. The information re-injection program processes the text one by one in the order in which it enters the queue, merges the current content to be changed with the content of the web page meta information library, and finally updates it to the web page meta information library.

[0102] It is worth noting that the reason why there is an independent information injection program instead of each process modifying the web page meta-information library at the same time is to ensure the atomicity and stability of the web page meta-information library modification operation.

[0103] like Figure 4As shown, a system for breadth data collection includes: a web page meta information library, an aggregation module, a quota allocation module, a link selection module, a downloader, a data post-processing and data cleaning module, and a re-injection queue module; the URL meta information in the web page meta information library is aggregated twice by the aggregation module to obtain a third pattern set, the quota allocation module determines the scheduling unit corresponding to each third pattern set, and allocates the daily download quota of the scheduling unit to each scheduling unit. The link selection module selects the link of the corresponding quota from the undownloaded link set corresponding to the third pattern set according to the daily download quota, and the downloader completes the download task, performs preliminary diversion analysis and refined data cleaning on the downloaded text content, and uses multiple quality models of different dimensions to score, and pushes the URL of the web page and its corresponding quality score to the re-injection queue to realize the update of the web page meta information library.

[0104] The linkbase is the core database of the system, which uses a standardized URL as a key to store URL meta-information related to the implementation of the method, including URLs that have been downloaded and URLs that have not been downloaded.

[0105] The aggregation module is used to process downloaded links and non-downloaded links, and obtain a third pattern set through two aggregations. In a specific embodiment, the implementation method includes: firstly, dividing the downloaded links in the web page meta information library into two parts, a prefix and a suffix, enumerating the sets of all inclusion relationships of the prefix and the suffix, and calculating the Cartesian product of the two sets to obtain a first pattern set; then, performing a first aggregation according to the pattern set according to the corresponding relationship, and filtering the first pattern set; then, processing the non-downloaded links to generate a second pattern set; then obtaining the intersection of the second pattern set and the filtered first pattern set, and realizing the attribution of the non-downloaded links, and finally obtaining a third pattern set through a second aggregation.

[0106] The quota allocation module is used to determine the scheduling unit corresponding to each third mode set, and allocate the daily download quota of the scheduling unit in each scheduling unit. First, all mode sets are sorted in reverse order according to effectiveness in the scheduling unit to obtain the actual number of downloadable links of the mode set. Then, all mode sets allocated to the download quota of the scheduling unit are globally sorted in reverse order according to effectiveness, and the total download quota of the day is allocated to the actual number of downloadable links of the corresponding mode set from high to low according to effectiveness, so as to obtain the daily download quota allocated to each mode set.

[0107] The link selection module is used to determine the final download link. According to the daily download quota allocated to each mode set, a link with a corresponding quota is selected from the undownloaded link set corresponding to the third mode set as the final selection result.

[0108] The downloader is used to execute the download task. During the download process, download control needs to be implemented, and the links selected every day are divided according to the scheduling units to which they belong. The links that need to be downloaded by each scheduling unit are distributed within a day for download according to the access pressure curve of the scheduling unit to which they belong.

[0109] The data post-processing and data cleaning module is used to perform preliminary diversion analysis and refined data cleaning on the downloaded text content, and use multiple quality models of different dimensions to score, and push the URL of the webpage and its corresponding quality score to the re-injection queue. The ones with higher quality scores are selected to enter the subsequent data processing flow to prepare for entering the large language model training corpus.

[0110] The refill queue module refers to a pre-deployed queue service that uses the first-in-first-out principle to update the web page meta information library. All information that has been post-processed and cleaned needs to be pushed to the refill queue before entering the web page meta information library, and then the web page meta information library is updated by a single information refill program.

[0111] The above description is a specific embodiment of the present invention and the technical principles used. If the changes made according to the concept of the present invention do not exceed the spirit covered by the description and drawings, they should still fall within the scope of protection of the present invention.

Claims

1. A method for collecting breadth data, characterized in that The following steps are involved: S1: Establishing web page meta information database; S2: Determine the download task parameters for the day, including the download quota for each scheduling unit and the total download quota for the day; S3: Select a corresponding number of links from the web page meta-information library and allocate download quotas; S4: download process control; S5: Download the text for post-processing and data cleaning, and the processed text enters the re-injection queue; S6: Update the web page meta-information library through information re-injection.

2. The method for collecting breadth data according to claim 1, characterized in that: The web page meta information library in S1 uses a standardized URL as a key to store URL meta information; the URL includes downloaded URLs and undownloaded URLs.

3. A method for collecting breadth data according to claim 1-2, characterized in that: The scheduling unit in S2 is one or a group of hosts. The scheduling unit download quota refers to setting a crawling limit for each scheduling unit every day; the total download quota is the total number of URLs that can be downloaded and processed in one day.

4. A method for collecting breadth data according to claims 1-3, characterized in that: The specific steps of S3 include: S301: Generate a first pattern set according to the downloaded link processing; S302: Filter the first pattern set to obtain a filtered first pattern set set; S303: Process the undownloaded links to generate a second pattern set; S304: Obtain the intersection of the second pattern set and the filtered first pattern set, and implement the attribution of the undownloaded links to obtain a third pattern set; S305: Determine the scheduling unit corresponding to each third mode set, and allocate a download quota to each scheduling unit; S306: Select a download link in each scheduling unit.

5. A method for collecting breadth data according to claims 1-4, characterized in that: The specific implementation of step S301 is to divide the downloaded link in the web page meta information library into two parts: a prefix and a suffix, then enumerate all sets of inclusion relationships between the prefix and the suffix, and calculate the Cartesian product of the two sets to obtain the first pattern set.

6. A method for collecting breadth data according to claims 1-4, characterized in that: In the step S302, a download link number threshold is preset, and pattern sets with download link numbers greater than the threshold and pattern sets at the top level of each domain are retained, and the remaining pattern sets are discarded to obtain a filtered first pattern set set.

7. A method for collecting breadth data according to claims 1-4, characterized in that: In step S303, the undownloaded links in the web page meta information library are divided into two parts: a prefix and a suffix. Then, all sets of inclusion relationships between the prefix and the suffix are enumerated, and the Cartesian product of the two sets is calculated to obtain a second pattern set.

8. A method for collecting breadth data according to claims 1-4, characterized in that: In step S303, the elements in the intersection are sorted from high to low according to the priority rule, and the link is assigned to the pattern set with higher priority; the priority is defined as: number of suffix-determined items > number of prefix-determined items > number of suffix-indetermined items > number of prefix-indetermined items > validity.

9. A method for collecting breadth data according to claims 1-8, characterized in that: The specific steps of S4 include: S401: Obtaining a website access pressure curve; S402: According to the pressure curve of the dispatching unit, adjustment is performed according to the formula: Where n is the total daily download volume, a is the control coefficient, which usually ranges from 1 to 2, and T i Delay for sharding time; S403: The downloader assigns tasks for downloading.

10. A method for collecting breadth data according to claims 1-9, characterized in that: The re-injection queue refers to a pre-deployed queue service that uses a first-in-first-out principle to implement updates to the web page meta-information library. All information that has undergone post-processing and data cleaning needs to be pushed to the re-injection queue before entering the web page meta-information library for update, and then a single information re-injection program is used to update the web page meta-information library.

11. A system for collecting breadth data, characterized in that include: Web page meta information base, aggregation module, quota allocation module, link selection module, downloader, data post-processing and data cleaning module, and refill queue module; The URL meta information in the web page meta information library is aggregated twice by the aggregation module to obtain a third pattern set. The quota allocation module determines the scheduling unit corresponding to each third pattern set, and allocates the daily download quota of the scheduling unit to each scheduling unit. The link selection module selects the link of the corresponding quota from the non-downloaded link set corresponding to the third pattern set according to the daily download quota. The download task is completed by the downloader, and the downloaded text content is subjected to preliminary diversion analysis and refined data cleaning, and is scored using quality models of multiple different dimensions. The URL of the web page and its corresponding quality score are pushed to the re-injection queue to realize the update of the web page meta information library.

Images