Webpage information processing method and system, electronic equipment and storage medium

An information processing method and webpage information technology, applied in the field of systems, electronic devices and storage media, and webpage information processing methods, can solve the problems that analysis modules and processing modules cannot be flexibly configured and modified, so as to save analysis and processing resources and improve The effect of processing efficiency

CN109597952AActive Publication Date: 2019-04-09江苏运满满信息科技有限公司
9 Cites 1 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2019-04-09

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides a webpage information processing method and system, electronic equipment and a storage medium, relates to the field of data processing, and is used for crawling and processing webpage resources. The webpage information processing method comprises the steps: acquiring a URL address to be crawled, and crawling webpage information according to the URL address; Calling an analysis module bound with the URL address in a database, analyzing the crawled webpage information, and extracting the data content of the webpage information; And calling a processing module bound with the URL address in a database, and processing the analyzed data content. According to the invention, the URL address is bound with the corresponding analysis module and processing module in a mutual mapping manner; when a crawling task of a certain URL address is executed, the URL address is transmitted in a process, so that a corresponding analysis module and a corresponding processing module are flexibly called from a database according to the URL address to process crawled webpage information; And the URL address and the corresponding analysis module and processing module can dynamically modify the configuration.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present application relates to the technical field of data processing, and in particular, relates to a webpage information processing method, system, electronic device and storage medium. Background technique

[0002] Crawler technology is a technology that automatically crawls specific web resources according to certain rules. Since the web pages of each website are different, custom development for specific web resources is required when crawling different web resources.

[0003] Existing crawler technology system reference figure 1 As shown, the scheduling center controls multiple crawling lines, and each crawling line corresponds to grabbing a webpage resource. Each crawling line includes three modules, namely the webpage download module, the webpage analysis module and the data processing module. The crawling lines are independent of each other. When adding a web page resource acquisition requirement, it is necessary to add a program for web...

Examples

Embodiment Construction

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals denote the same or similar structures in the drawings, and thus their repeated descriptions will be omitted.

[0038] The present application provides a method for processing webpage information that can dynamically modify configuration, and is flexible in parsing and processing, and saves resources, as well as a corresponding system, device, and storage medium. The following uses the Java language as an example to illustrate the web page information processing method of the present application. Those skilled in the art can also u...