Unlock instant, AI-driven research and patent intelligence for your innovation.

Network data acquisition and processing method and device and electronic equipment

A technology for collecting and processing network data, applied in network data retrieval, electronic digital data processing, network data browsing optimization, etc.

Pending Publication Date: 2021-05-07
北京鼎普科技股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The purpose of the embodiments of the present invention is to provide a network data collection and processing method, device and electronic equipment to solve the existing problems in data collection and storage

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Network data acquisition and processing method and device and electronic equipment
  • Network data acquisition and processing method and device and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] The implementation of the present invention will be illustrated by specific specific examples below, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0038] In the following description, for purposes of illustration rather than limitation, specific details, such as specific system architectures, interfaces, and techniques, are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the invention may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0039] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descrip...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a network data acquisition and processing method and device and electronic equipment. The method comprises the following steps: acquiring target network data; then generating a scheduling data file and a scheduling control file according to the target network data, wherein the scheduling data file is used for storing tasks needing to be collected, each record in the scheduling data file comprises a data length and data information, and the scheduling control file is used for controlling the scheduling data file and comprises a data source, a priority level and data reading related information; and controlling the data information in the scheduling data file to be analyzed and stored in a file queue through the scheduling control file. According to the invention, network data acquisition and storage efficiency is high, and the resource utilization rate is high.

Description

technical field [0001] The embodiments of the present invention relate to the field of network data collection, and in particular to a network data collection and processing method, device and electronic equipment. Background technique [0002] When collecting network data, multi-tasking is required to collect data from multiple sites, and distributed data is often used to improve data collection efficiency, that is, one collection scheduler and multiple collection crawlers to achieve simultaneous collection of multiple site tasks. [0003] In order to realize network data collection, it is necessary to select more important and out-of-degree URLs in the site as the entry addresses of the collected websites (called seed URLs). The crawler will start collecting from these seed URLs. After the web page data is collected, it needs to be parsed again Data elements in the page, extract the URL in the page and collect again. Such a URL can be parsed to generate a batch of new URL...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50G06F9/54G06F16/955G06F16/957G06F16/906
CPCG06F9/4881G06F9/5027G06F9/546G06F16/955G06F16/9574G06F16/906G06F2209/484G06F2209/548
Inventor 刘龙强
Owner 北京鼎普科技股份有限公司