Webpage information collection system and method based on crawler technology, storage medium and equipment

Abstract

The invention discloses a webpage information collection system and method based on a crawler technology, a storage medium and equipment. The webpage information collection system comprises a URL sequence grabber for grabbing a URL sequence in the Internet according to a search task by utilizing a crawler strategy in a crawler strategy device; the URL manager is provided with a crawler portrait wide table, and the key information in the URL sequence capturing process is recorded into the crawler portrait wide table; the crawler strategy adjuster is used for dynamically adjusting a crawler strategy according to key information recorded in the crawler portrait wide table; the webpage downloader is used for downloading webpage information from the captured URL sequence; the webpage analyzer is used for analyzing the downloaded webpage information and removing repeated webpage information through a star large model; and the data memory is used for storing the webpage information without duplication. According to the invention, the collected webpage information has the characteristics of wide coverage range, high quality and low repeatability.

Description

Technical Field

[0001] The present invention relates to the technical field of web page information collection, and in particular to a web page information collection system, method, storage medium and device based on crawler technology. Background Art

[0002] In the digital age, network data has become a core resource for business decision-making, academic research, public services, and other fields. As an efficient and automated data acquisition tool, crawler technology is increasingly being used. Crawlers can cover vast amounts of network data, rapidly capturing thousands or even millions of data items. This overcomes the scale limitations of manual data collection and significantly reduces manual collection time. Crawlers can replace humans in repetitive, high-intensity data collection tasks, saving labor costs.

[0003] Crawling technology collects webpage information by fetching URL sequences to obtain webpage content. Modern websites widely use JavaScript to dynamically generate content. However, traditional crawling technology uses a fixed crawling strategy that cannot adapt to changes in dynamic webpages and content. It also cannot discover new URL sequences through links within the page, resulting in incomplete coverage of collected webpage information. Secondly, the URL sequence fetching process may contain invalid or outdated links, resulting in low-quality webpage information. Furthermore, crawling URL sequences using crawling technology generates a large number of URL sequences, making efficient deduplication and management a challenge. Improper management can lead to duplicate crawling, wasted resources, and even blockage by the target website. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present invention provides a web page information collection system, method, storage medium and device based on crawler technology, so that the collected web page information has the characteristics of wide coverage, high quality and low duplication.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solutions: a web page information collection system based on crawler technology, comprising: a URL sequence grabber, a URL sequence manager, a crawler policy device, a web page downloader, a web page parser and a data storage device; The URL sequence crawler uses the crawler strategy in the crawler strategy device to crawl URL sequences in the Internet according to the search task; The URL manager is provided with a crawler portrait wide table, which records key information in the process of crawling URL sequences into the crawler portrait wide table; The crawler strategy adjuster dynamically adjusts the crawler strategy according to the key information recorded in the crawler portrait width table; The web page downloader is used to download web page information from the captured URL sequence; The web page parser is used to parse the downloaded web page information and remove duplicate web page information through the Xingchen big model; The data storage device is used to store deduplicated web page information.

[0006] Furthermore, the crawler portrait wide table records three columns of key information, including: field name, field type and field description of key information; the key information includes: search task, crawled URL sequence, domain name, crawling time, web page status information, web page size, crawled depth information, web page information hash value, web page importance, web page coverage and web page timeliness.

[0007] Furthermore, the process of dynamic adjustment of the crawler strategy is as follows: i. Adjust the crawler strategy by comparing the scores of web page importance, web page coverage, and web page timeliness; ii. Use IPV9 technology to remove invalid or outdated URL sequences from the captured URL sequences.

[0008] Furthermore, when the score of web page importance > the score of web page coverage > the score of web page timeliness, a non-complete Pagerank strategy is adopted; when the score of web page importance > the score of web page timeliness > the score of web page coverage, a large site priority strategy is adopted; when the score of web page coverage is the highest or the score of web page timeliness is the highest, a breadth-first traversal strategy is adopted.

[0009] Furthermore, The importance rating of the webpage in question ,in, represents the damping coefficient, n represents the number of web pages pointing to the crawled web page, i represents the index of n, represents the PageRank value of the i-th web page, represents the number of outbound links of the i-th web page; The web page coverage score is obtained by the ratio of the number of crawled web pages to the total number of web pages on the Internet; The web page timeliness score is obtained by calculating the similarity between the captured web page information and the actual information of the current web page.

[0010] Furthermore, the specific process of removing invalid or outdated URL sequences from the captured URL sequence using IPV9 technology is as follows: The Internet encodes the URL sequence and its metadata hash and expiration time into the IPv9 address; When accessing the corresponding web page through the crawler strategy, the metadata in the IPV9 address is extracted and the hash value of the web page information is calculated based on the metadata; If the calculated web page hash value is consistent with the metadata hash value, and the crawling time does not exceed the expiration time, the corresponding URL sequence is crawled; otherwise, it is marked as an invalid URL sequence and no longer crawled.

[0011] Furthermore, the specific process of removing duplicate web page information through the Xingchen Big Model is as follows: i. Input the parsed web page information into the Xingchen Big Model to extract the key text content, filter out repeated HTML tags, special characters and spaces in the key text content, and obtain pure text; ii. Input the clean text into the Xingchen model to generate semantic vectors; iii. Calculate the cosine similarity between the semantic vectors of different web pages. If the cosine similarity exceeds the set threshold, it means that the web page information is repeated; iv. Cluster the duplicate web page information, filter out representative web page information, and delete other web page information.

[0012] Furthermore, the present invention also provides a web page information collection method using the web page information collection system based on crawler technology, comprising the following steps: Step S1: crawling URL sequences using crawler strategies according to the search task; Step S2: Record key information in the process of crawling URL sequences into the crawler portrait wide table; Step S3: Download web page information from the captured URL sequence, parse it, remove duplicate web page information through the Xingchen big model, and save it; Step S4: Dynamically adjust the crawler strategy according to the information recorded in the crawler portrait wide table, repeat steps S1 to S3 until the termination condition is reached, and complete the web page information collection process.

[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the web page information collection method based on crawler technology.

[0014] Furthermore, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the web page information collection method based on crawler technology is implemented.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention records key information in the process of crawling URL sequences into a wide crawler portrait table, and associates the crawled URL sequences with the configuration information and running status information of the crawler, so as to fully understand the behavior and performance of the crawler when crawling URL sequences, and provide a reliable basis for subsequent crawler strategy adjustments; at the same time, the crawler portrait wide table can be used to trace the source of the crawled URL sequences, troubleshoot problems, and locate them; (2) The present invention adjusts the crawler strategy by comparing the scores of web page importance, web page coverage, and web page timeliness, which can adapt to the changes in dynamic web pages and content, making the crawler work more intelligently and efficiently, better meeting the data crawling needs in different business scenarios, and improving data quality and crawling efficiency; (3) The present invention uses IPV9 technology to remove invalid or outdated URL sequences from the captured URL sequence, removes invalid URL sequences by comparing the consistency of the hash values ​​in the IPV9 addresses before and after the URL sequence is captured, and removes outdated URL sequences by comparing the expiration time of the URL sequence with the capture time. By utilizing the characteristics of the IPV9 address, invalid or outdated URL sequences can be quickly screened out, thereby improving data processing efficiency; (4) The present invention removes duplicate web page information through the Xingchen Big Model. The Xingchen Big Model has powerful text analysis and pattern recognition capabilities, can deeply understand the semantics and structure of web page content, and remove duplicate web page information through the semantic similarity of web page content, reducing the pressure of redundant data storage and improving the retrieval quality of web page information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a web page information collection system based on crawler technology according to the present invention; Figure 2 A schematic diagram of the crawler strategy adjustment in the present invention; Figure 3 The flowchart of the web page information collection method based on crawler technology of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be further explained below with reference to the accompanying drawings.

[0018] like Figure 1 This is a schematic diagram of a web page information collection system based on crawler technology according to the present invention, which includes: URL sequence grabber, URL sequence manager, crawler policy maker, web page downloader, web page parser and data storage; The URL sequence crawler uses the crawler strategy in the crawler strategy device to crawl URL sequences on the Internet according to the search task. It can accurately crawl URL sequences from the Internet according to the crawler strategy, providing a basis for the accurate collection of web page information. The URL manager has a crawler portrait wide table, which records key information during the crawling of URL sequences, thereby tracing the crawled URL sequences and troubleshooting problems. The crawler strategy adjuster dynamically adjusts the crawler strategy based on the key information recorded in the crawler portrait wide table, thereby improving the stability and success rate of the crawler. It can be widely applied in various application scenarios. The web page downloader is used to download web page information from the captured URL sequence. It has efficient network request and response processing capabilities and can quickly and stably obtain web page resources. The web page parser is used to parse the downloaded web page information, remove duplicate web page information through the Xingchen big model, avoid web page information redundancy, and ensure the diversity of web page information; The data storage device is used to store the deduplicated web page information.

[0019] The web page information collection system based on crawler technology of the present invention has the characteristics of wide coverage, high quality and low duplication of the collected web page information, and the division of labor and cooperation among various modules, from crawling and managing URL sequences to downloading, parsing and storing web page information. The entire process does not require human intervention, which greatly improves the work efficiency of web page information collection and makes the collected web page information have the characteristics of wide coverage, high quality and low duplication.

[0020] As shown in Table 1, the crawler portrait wide table in the present invention records three columns of key information, including: field name, field type and field description of key information; wherein, key information includes: search task, captured URL sequence, domain name, capture time, web page status information, web page size, captured depth information, web page information hash value, web page importance, web page coverage and web page timeliness. The present invention associates the captured URL sequence with the crawler's configuration information and running status information, so as to fully understand the crawler's behavior and performance when capturing URL sequences, and provide a reliable basis for subsequent crawler strategy adjustments; at the same time, the crawler portrait wide table can be used to trace the captured URL sequence, troubleshoot and locate problems.

[0021] Table 1 Reptile portrait wide table

[0022] The present invention adjusts the crawler strategy by comparing the scores of web page importance, web page coverage and web page timeliness. Adjusting the crawler strategy can adapt to changes in dynamic web pages and content, making the crawler work more intelligently and efficiently, better meeting the data crawling needs in different business scenarios, and improving data quality and crawling efficiency. Specifically, when the score of web page importance > the score of web page coverage > the score of web page timeliness, the non-complete Pagerank strategy is adopted, which can more accurately locate those pages pointed to by many high-quality web page links. These pages often contain more valuable content, and because the strategy focuses on important web pages, the crawler can concentrate on crawling and storing the most valuable information under limited resources, avoiding wasting resources on a large number of low-value web pages; when the score of web page importance > the score of web page timeliness > the score of web page coverage, the large site priority strategy is adopted. Large websites usually have more complete technical architecture and content review mechanisms, and the information they publish is relatively more reliable and accurate. Using the large site priority strategy, the crawler gives priority to crawling web pages of large websites, and can obtain higher quality and more timely data; when the score of web page coverage is the highest or the score of web page timeliness is the highest, the breadth-first traversal strategy is adopted to ensure that all web pages of the target website are covered as much as possible, thereby improving the comprehensiveness of the information. At the same time, the breadth-first traversal strategy can quickly traverse the target website and promptly discover and crawl newly released web pages. Because this strategy prioritizes processing all pages in the current tier before moving on to the next tier, it captures the latest content on the site more quickly.

[0023] Web page importance rating ,in, represents the damping coefficient, n represents the number of web pages pointing to the crawled web page, i represents the index of n, represents the PageRank value of the i-th web page, represents the number of outbound links of the i-th web page; The web page coverage score is obtained by the ratio of the number of crawled web pages to the total number of web pages on the Internet. The higher the ratio, the higher the web page coverage; The timeliness score of a web page is obtained by calculating the similarity between the captured web page information and the actual information of the current web page. The higher the similarity, the stronger the timeliness of the web page.

[0024] The present invention uses IPV9 technology to remove invalid or outdated URL sequences from the captured URL sequence, specifically: The Internet encodes the URL sequence and its metadata hash and expiration time into the IPv9 address; When accessing the corresponding web page through the crawler strategy, the metadata in the IPV9 address is extracted and the hash value of the web page information is calculated based on the metadata; If the calculated web page hash value is consistent with the metadata hash value, and the crawling time does not exceed the expiration time, the corresponding URL sequence is crawled; otherwise, it is marked as an invalid URL sequence and no longer crawled.

[0025] The present invention removes invalid URL sequences by comparing the consistency of hash values ​​in IPV9 addresses before and after crawling URL sequences, removes outdated URL sequences by comparing the expiration time of URL sequences with the crawling time, and quickly filters out invalid or outdated URL sequences by utilizing the characteristics of IPV9 addresses, thereby improving data processing efficiency.

[0026] The Xingchen Big Model has powerful text analysis and pattern recognition capabilities, and can deeply understand the semantics and structure of web page content. This invention uses the Xingchen Big Model to remove duplicate web page information, which can reduce the pressure of redundant data storage and improve the retrieval quality of web page information. The specific process is as follows: i. Input the parsed web page information into the Xingchen Big Model to extract the key text content, filter out repeated HTML tags, special characters and spaces in the key text content, and obtain pure text; ii. Input the clean text into the Xingchen model to generate semantic vectors; iii. Calculate the cosine similarity between the semantic vectors of different web pages. If the cosine similarity exceeds the set threshold, it means that the web page information is repeated; iv. Cluster the duplicate web page information, filter out representative web page information, and delete other web page information.

[0027] like Figure 3 In one technical solution of the present invention, a web page information collection method based on crawler technology is also provided, comprising the following steps: Step S1: crawling URL sequences using crawler strategies according to the search task; Step S2: Record key information in the process of crawling URL sequences into the crawler portrait wide table; Step S3: Download web page information from the captured URL sequence, parse it, remove duplicate web page information through the Xingchen big model, and save it; Step S4: Dynamically adjust the crawler strategy according to the information recorded in the crawler portrait wide table, repeat steps S1 to S3 until the termination condition is reached, and complete the web page information collection process.

[0028] In one technical solution of the present invention, a computer-readable storage medium is further provided, storing a computer program, wherein the computer program enables a computer to execute a web page information collection method based on crawler technology.

[0029] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a web page information collection method based on crawler technology is implemented.

[0030] In the embodiments disclosed herein, computer storage media may be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0031] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0032] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A web page information collection system based on crawler technology, characterized in that: include: URL sequence grabber, URL sequence manager, crawler policy maker, web page downloader, web page parser and data storage; The URL sequence crawler uses the crawler strategy in the crawler strategy device to crawl URL sequences in the Internet according to the search task; The URL manager is provided with a crawler portrait wide table, which records key information in the process of crawling URL sequences into the crawler portrait wide table; The crawler strategy adjuster dynamically adjusts the crawler strategy according to the key information recorded in the crawler portrait width table; The web page downloader is used to download web page information from the captured URL sequence; The web page parser is used to parse the downloaded web page information and remove duplicate web page information through the Xingchen big model; The data storage device is used to store deduplicated web page information.

2. A webpage information collection system based on crawler technology according to claim 1, characterized in that: The crawler portrait wide table records key information in three columns, including: field name, field type and field description of key information; the key information includes: search task, crawled URL sequence, domain name, crawling time, web page status information, web page size, crawled depth information, web page information hash value, web page importance, web page coverage and web page timeliness.

3. A web page information collection system based on crawler technology according to claim 2, characterized in that: The process of dynamic adjustment of the crawler strategy is as follows: i. Adjust the crawler strategy by comparing the scores of web page importance, web page coverage, and web page timeliness; ii. Use IPV9 technology to remove invalid or outdated URL sequences from the captured URL sequences.

4. A webpage information collection system based on crawler technology according to claim 3, characterized in that: When the score of web page importance > the score of web page coverage > the score of web page timeliness, the non-complete Pagerank strategy is adopted; when the score of web page importance > the score of web page timeliness > the score of web page coverage, the large site priority strategy is adopted; when the score of web page coverage is the highest or the score of web page timeliness is the highest, the breadth-first traversal strategy is adopted.

5. The webpage information collection system based on crawler technology according to claim 3 is characterized in that: The importance rating of the webpage in question ,in, represents the damping coefficient, n represents the number of web pages pointing to the crawled web page, i represents the index of n, represents the PageRank value of the i-th web page, represents the number of outbound links of the i-th web page; The web page coverage score is obtained by the ratio of the number of crawled web pages to the total number of web pages on the Internet; The web page timeliness score is obtained by calculating the similarity between the captured web page information and the actual information of the current web page.

6. The webpage information collection system based on crawler technology according to claim 3, characterized in that: The specific process of removing invalid or outdated URL sequences from the captured URL sequence using IPV9 technology is as follows: The Internet encodes the URL sequence and its metadata hash and expiration time into the IPv9 address; When accessing the corresponding web page through the crawler strategy, the metadata in the IPV9 address is extracted and the hash value of the web page information is calculated based on the metadata; If the calculated hash value of the webpage is consistent with the hash value of the metadata, and the crawling time does not exceed the expiration time, crawl the corresponding URL sequence; Otherwise, it is marked as an invalid URL sequence and will not be crawled again.

7. The webpage information collection system based on crawler technology according to claim 1, characterized in that: The specific process of removing duplicate web page information through the Xingchen Big Model is as follows: i. Input the parsed web page information into the Xingchen Big Model to extract the key text content, filter out repeated HTML tags, special characters and spaces in the key text content, and obtain pure text; ii. Input the clean text into the Xingchen model to generate semantic vectors; iii. Calculate the cosine similarity between the semantic vectors of different web pages. If the cosine similarity exceeds the set threshold, it means that the web page information is repeated; iv. Cluster the duplicate web page information, filter out representative web page information, and delete other web page information.

8. A web page information collection method using the web page information collection system based on crawler technology according to any one of claims 1 to 7, characterized in that: The steps include: Step S1: crawling URL sequences using crawler strategies according to the search task; Step S2: Record key information in the process of crawling URL sequences into the crawler portrait wide table; Step S3: Download web page information from the captured URL sequence, parse it, remove duplicate web page information through the Xingchen big model, and save it; Step S4: Dynamically adjust the crawler strategy according to the information recorded in the crawler portrait wide table, repeat steps S1 to S3 until the termination condition is reached, and complete the web page information collection process.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables the computer to execute the web page information collection method based on crawler technology as claimed in claim 8.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the web page information collection method based on crawler technology as claimed in claim 8 is implemented.

Images