Automatic webpage collection task sorting method based on URL rule matching

A technology for collecting tasks and rules, applied in the field of big data, can solve problems such as increasing the difficulty of data collection, analysis and statistics, and achieve the effects of improving management capabilities, enhancing throughput and efficiency, and enhancing throughput

Inactive Publication Date: 2020-10-30
浪潮卓数大数据产业发展有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, with the rapid growth of application scale, it also increases the difficulty of data collection, analysis and statistics of online economic activities

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Automatic webpage collection task sorting method based on URL rule matching

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0029] combined with figure 1 , the present embodiment proposes an automatic sorting method for web page collection tasks based on URL rule matching, which includes three stages: 1) matching rule configuration, 2) task scheduling and arrangement, and 3) hierarchical topic model construction.

[0030] 1) Matching rule configuration stage:

[0031] In the collection task, formulate regular rules for url pattern matching of web page collection tasks, and configure regular rules to be downloaded, regular rules for parsing and regular rules for content extraction according to the stage to which the task belongs.

[0032] At this stage,

[0033] 1.1) When configuring regular rules to be downloaded according to the stage to which the task belongs, use a regular expression to set the url of the link to be downloaded in the matching task queue. The regular download rule takes the task entry url address link as the rule matching mode and formulates according to the entry url link The ...

Embodiment 2

[0049] Based on Embodiment 1, this embodiment proposes an automatic sorting method for webpage collection tasks based on URL rule matching. This method can also set the collection task cycle configuration time, and the collection tasks after startup are sent to the corresponding website task queue in redis , the task scheduling node pulls the collection task from the task queue for segmentation and analysis, writes the task into the kafka download topic, and the collection node downloader obtains the task download link from kafka and downloads the link according to the url of the task website entry. After the download is successful task.

[0050] To sum up, it can be seen that the automatic sorting method of webpage collection tasks based on URL rule matching of the present invention can effectively reduce the scheduling time of collection tasks, realize fully automatic task flow and processing of collection tasks without manual intervention, and enhance task processing at the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a webpage acquisition task automatic sorting method based on URL rule matching, and relates to the technical field of big data, and the method comprises the steps: 1) matchingrule configuration: formulating a webpage acquisition task url mode matching regular rule in an acquisition task, and configuring a to-be-downloaded, analysis and content extraction regular rule according to the stage to which the task belongs; 2) scheduling and arranging tasks: pulling a to-be-executed task theme, performing unpacking according to the scheduling distribution queue and placing thetask packets in a to-be-processed task queue, enabling the rule matching nodes to sequentially acquire task packets from the task queue, carrying out the forwarding after matching with the rule matching nodes according to url address modes of the task packets, downloading, analyzing and extracting rules as task scheduling sorters, realizing one-to-one correspondence with kafka topics, and callingproducer interfaces to write the matched tasks into a topic domain; 3) constructing a hierarchical topic model based on task scheduling and processing analysis and configuring a sorting strategy. Themethod can reduce task scheduling time and enhance the throughput of task processing.

Description

technical field [0001] The invention relates to the field of big data technology, in particular to an automatic sorting method for web page collection tasks based on URL rule matching. Background technique [0002] With the rapid development of the Internet and big data technology, more and more economic activities have begun to expand online channels, and the online economy has become an important part of economic development. In particular, industries represented by retail, entertainment, finance, and life services, driven by big data technology, are constantly bursting with creative vitality, and the products they have spawned have completely changed all aspects of social life. However, with the rapid growth of application scale, it also increases the difficulty of data collection, analysis and statistics of online economic activities. [0003] Based on this, aiming at the online data collection tasks of distributed web applications based on modern big data technology, a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/955G06F9/50G06F9/54H04L29/08
CPCG06F9/5027G06F9/546G06F2209/548G06F16/9566H04L67/06
Inventor王勇
Owner浪潮卓数大数据产业发展有限公司