Web page information processing method, system, electronic device and storage medium

Abstract

The invention provides a webpage information processing method and system, electronic equipment and a storage medium, relates to the field of data processing, and is used for crawling and processing webpage resources. The webpage information processing method comprises the steps: acquiring a URL address to be crawled, and crawling webpage information according to the URL address; Calling an analysis module bound with the URL address in a database, analyzing the crawled webpage information, and extracting the data content of the webpage information; And calling a processing module bound with the URL address in a database, and processing the analyzed data content. According to the invention, the URL address is bound with the corresponding analysis module and processing module in a mutual mapping manner; when a crawling task of a certain URL address is executed, the URL address is transmitted in a process, so that a corresponding analysis module and a corresponding processing module are flexibly called from a database according to the URL address to process crawled webpage information; And the URL address and the corresponding analysis module and processing module can dynamically modify the configuration.

Description

technical field

[0001] The present application relates to the technical field of data processing, and in particular, to a web page information processing method, system, electronic device and storage medium. Background technique

[0002] Crawler technology is a technology that automatically crawls specific webpage resources according to certain rules. Since the webpages of each website are different, it is necessary to customize development for specific webpage resources when crawling different webpage resources.

[0003] Existing crawler technology system reference figure 1 As shown, the dispatch center controls multiple crawling lines, each crawling line corresponds to crawling a webpage resource, and each crawling line includes 3 modules, namely the webpage downloading module, the webpage parsing module and the data processing module. The crawling lines are independent of each other. When the acquisition requirement of a web page resource is increased, a program for web...

Examples

Embodiment Construction

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repeated descriptions will be omitted.

[0038] The present application provides a web page information processing method, a corresponding system, a device and a storage medium, which can dynamically modify the configuration, have flexible parsing and processing, and save resources. The following uses Java language as an example to describe the web page information processing method of the present application. Those skilled in the art can also use other programmi...

technical field

[0001] The present application relates to the technical field of data processing, and in particular, to a web page information processing method, system, electronic device and storage medium. Background technique

[0002] Crawler technology is a technology that automatically crawls specific webpage resources according to certain rules. Since the webpages of each website are different, it is necessary to customize development for specific webpage resources when crawling different webpage resources.

[0003] Existing crawler technology system reference figure 1 As shown, the dispatch center controls multiple crawling lines, each crawling line corresponds to crawling a webpage resource, and each crawling line includes 3 modules, namely the webpage downloading module, the webpage parsing module and the data processing module. The crawling lines are independent of each other. When the acquisition requirement of a web page resource is increased, a program for web...

Examples

Embodiment Construction

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repeated descriptions will be omitted.

[0038] The present application provides a web page information processing method, a corresponding system, a device and a storage medium, which can dynamically modify the configuration, have flexible parsing and processing, and save resources. The following uses Java language as an example to describe the web page information processing method of the present application. Those skilled in the art can also use other programmi...

Images