Unlock instant, AI-driven research and patent intelligence for your innovation.

Network data capturing method based on big data

A network data and big data technology, applied in network data indexing, network data retrieval, other database retrieval and other directions, can solve the problems of unable to know the signature algorithm, unable to capture the data content of mobile APP, not intelligent enough, etc.

Pending Publication Date: 2020-07-14
安徽火蓝数据有限公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, the inventor found that when the mobile terminal APP communicates with the server, since the request communication data packet usually contains a lot of parameter signatures, if the signature algorithm of these parameters cannot be known, the crawler often cannot know the signature algorithm of these parameters, resulting in It is impossible to simulate the request of the mobile terminal APP to communicate with the server, so it is impossible to capture the data content in the mobile APP
In addition, the current mobile terminal APP often pushes to the user according to the current news hotspots, but there is currently no method for automatically grabbing news hotspots, and manual configuration of grabbing rules is often required, which is not smart enough

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Network data capturing method based on big data
  • Network data capturing method based on big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] Through the description of the embodiments below, the specific implementation of the present invention includes the shape, structure, mutual position and connection relationship between the various parts, the function and working principle of each part, the manufacturing process and the operation and use method of the various components involved. etc., to make further detailed descriptions to help those skilled in the art have a more complete, accurate and in-depth understanding of the inventive concepts and technical solutions of the present invention.

[0037] In order to realize the purpose of the above invention, such as figure 1 As shown, the present invention provides a method for grabbing network data based on big data, including

[0038] S10, configuring the listening terminal as a proxy server;

[0039] S20, the target APP sends communication data to the target server through the proxy server;

[0040] S30. The proxy server simulates the target APP to send co...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a network data capturing method based on big data. The network data capturing method comprises that: a monitoring terminal is configured as a proxy server; a target APP sends communication data to a target server through the proxy server; the proxy server simulates the target APP to send communication data to the target server; the proxy server obtains a target field according to big data analysis; and a capture rule is configured, and the proxy server captures data sent by the target server according to the target field. According to the method, the monitoring terminalis configured to be the proxy server, the proxy server simulates the target APP to send the communication data to the target server, the capture rule is configured after big data analysis or the target field is passed, and the proxy server captures the data sent by the target server according to the target field. Therefore, network news hotspots can be automatically captured, manual configurationis not needed, and high efficiency and intelligence are achieved.

Description

technical field [0001] The invention relates to the technical field of data capture, in particular to a network data capture method based on big data. Background technique [0002] At present, with the rapid development of the mobile Internet, mobile terminal APP (Application, application program) has become the main battlefield for people to surf the Internet, so there is a greater demand for data capture of mobile terminal APP, such as Sina APP, Tencent News APP, Baidu Data capture in news apps such as APP and Toutiao APP. [0003] At present, the frameworks for data capture mainly include WebCollector, Nutch, PySpider, WebMagic, etc. The existing crawling method is to directly use the URL of the web page as the entry address. [0004] However, the inventor found that when the mobile terminal APP communicates with the server, since the request communication data packet usually contains a lot of parameter signatures, if the signature algorithm of these parameters cannot b...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951G06F8/53H04L29/08
CPCG06F16/951G06F8/53H04L67/56
Inventor 张俊杰耿雁萍
Owner 安徽火蓝数据有限公司