Data storage method and device

A data storage device and data storage technology, which is applied in the field of data transmission, can solve problems such as user inability to use, data errors, etc., and achieve the effect of avoiding complex problems and improving flexibility

CN110955853APending Publication Date: 2020-04-03BEIJING GRIDSUM TECH CO LTD
8 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Current Assignee / Owner
Publication Date
2020-04-03

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a data storage method and device, and aims to solve the problem that a user cannot use data due to errors of the data uploaded by a crawler end. The method comprises the stepsof obtaining a confirmation submission request, wherein the confirmation submission request is used for indicating that a crawler end crawls data required by a crawling task; in response to the confirmation submission request, reading a plurality of partial data from a first storage area, the first storage area storing data crawled by the crawler end according to the crawling task; and obtaining data required by the crawling task according to the plurality of partial data, and storing the data required by the crawling task.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of data transmission, in particular to a data storage method and device. Background technique

[0002] A web crawler, also known as a web spider or a web robot, is a program or script that automatically grabs information on the World Wide Web according to certain rules.

[0003] In the process of crawling data, the crawler crawls according to the crawling task. After crawling the data according to the crawling task, the crawler will upload the data for the user to use. However, there may be dependencies between the data required in the crawling task. The data uploaded by the crawler at one time may not be the complete data required in the crawling task, or the crawler has an error during the crawling process. The wrong data fetched is also uploaded, which will affect the user's use of data. Contents of the invention

[0004] In view of the above problems, the purpose of the embodiments of the present invention is ...

Examples

Embodiment Construction

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are some, but not all, embodiments of the present invention. For those skilled in the art, other embodiments can also be obtained based on these embodiments without making creative efforts.

[0026] An embodiment of the present invention provides a data storage system, figure 1 It is a schematic diagram of the data storage system architecture in the embodiment of the present invention, see figure 1 As shown, the system includes: a crawler end 101 and a server end 102 .

[0027] Wherein, the crawler end may be a tool with a data crawling function, such as a web crawler. The server can be a server capable of receiving, storing and processing data, for example: a data receiving server, that is, a server, and the data receiving server can receive the crawled raw data transmi...