Unlock instant, AI-driven research and patent intelligence for your innovation.

Data acquisition method, system and device based on multi-channel agent

A data acquisition, multi-channel technology, applied in the field of network security, to achieve the effect of accurate and reliable data, convenient management and query

Pending Publication Date: 2021-03-19
NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Sometimes people need to continuously obtain the required data from a specific website, and the website usually prevents others from obtaining the information of their own website in batches by intercepting the client request IP

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data acquisition method, system and device based on multi-channel agent
  • Data acquisition method, system and device based on multi-channel agent

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly illustrate the spirit of the content disclosed in the application with the accompanying drawings and detailed descriptions. After any person skilled in the art understands the embodiments of the content of the application , when it can be changed and modified by the technology taught in the content of the application, it does not depart from the spirit and scope of the content of the application.

[0042] The exemplary embodiments and descriptions of the present application are used to explain the present application, but not to limit the present application. In addition, elements / members with the same or similar numbers used in the drawings and embodiments are used to represent the same or similar parts.

[0043] The terms "first", "second", ... etc. used herein do not specifically refer to a sequence or order, nor are they...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data acquisition method, system and device based on a multi-channel agent. The data acquisition method comprises the following steps: acquiring keywords in required data; calling a distributed crawler framework according to the acquired keywords, wherein a plurality of crawler nodes are arranged in the distributed crawler framework; changing the IPs of the crawler nodes; receiving webpage data crawled by crawler nodes with different IPs; performing data extraction, conversion and loading processing on the received webpage data to obtain valid data; and storing the valid data. The crawler nodes in the distributed crawler framework form the multi-channel agent, the IPs of the crawler nodes in the distributed crawler framework are changed before crawling information each time, an anti-crawler detection means of a website can be provided by bypassing data, and the crawler nodes are used for achieving operations such as multi-channel agent, intelligence discovery, cleaning and conversion, and ensuring accuracy and reliability of data.

Description

technical field [0001] The present application belongs to the technical field of network security, and in particular relates to a multi-channel agent-based data acquisition method, system and device. Background technique [0002] With the rapid development of science and technology, human society has entered the era of big data. The emergence and development of the Internet makes it relatively easy for people to obtain and utilize globalized data. Sometimes people need to continuously obtain the required data from a specific website, and the website usually prevents others from obtaining the information of their own website in batches by intercepting the IP address requested by the client. Contents of the invention [0003] In order to overcome the problems existing in related technologies at least to a certain extent, the present application provides a data acquisition method, system and device based on a multi-channel agent. [0004] According to the first aspect of th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951G06F16/9535G06F16/22G06F16/25
CPCG06F16/951G06F16/9535G06F16/254G06F16/22
Inventor 张凯曹首峰贺欣李海灵王啸李晓倩贺龙涛邬崇晖
Owner NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT