Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method, device, system, server and computer storage medium

A data processing and data storage system technology, applied in the field of systems, devices, servers, computer storage media, and data processing methods, can solve problems such as slow data backup speed, incomplete data, and complicated processes, so as to avoid incomplete data, Improve safety and save time

Active Publication Date: 2021-02-23
BEIJING QIHOO TECH CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Backing up data from each child node to the local child node will cause the following problems: the data is stored locally in each child node discretely, if a child node goes down, the data will be incomplete; if the discrete data is copied out, the centralized Storage, once all data needs to be restored, the data must be copied to the local directory according to the corresponding path, the process is complicated
[0005] However, when data backup is performed through the master node, there will be problems such as slow data backup speed and poor performance.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device, system, server and computer storage medium
  • Data processing method, device, system, server and computer storage medium
  • Data processing method, device, system, server and computer storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] figure 1 A schematic flowchart of a data processing method according to Embodiment 1 of the present invention is shown. This method is applied to a data storage system including a master node and multiple sub-nodes, and the method is executed in parallel by each sub-node, such as figure 1 As shown, the method includes the following steps:

[0031] Step S100, according to the data backup request, start the data backup service for connecting the child node and the intermediate storage system.

[0032] Wherein, the data backup service is pre-configured with a designated path for backing up data to the intermediate storage system.

[0033] Among them, the intermediate storage system is used to store the data backed up by each child node. It is a storage system independent of the data storage system. The intermediate storage system is a distributed file system with large bandwidth, large capacity, and large I / O throughput. Advantages, therefore, each child node of the dat...

Embodiment 2

[0040] figure 2 A schematic flowchart of a data processing method according to Embodiment 2 of the present invention is shown. This method is applied to a data storage system including a master node and multiple sub-nodes, and the method is executed by each sub-node in parallel, such as figure 2 As shown, the method includes the following steps:

[0041] Step S200, according to the data backup request, start the data backup service for connecting the child node and the intermediate storage system.

[0042] Among them, the intermediate storage system includes: HDFS system. The HDFS system has the advantages of large bandwidth, large capacity, and large I / O throughput. Therefore, it can support each sub-node of the data storage system to back up data to the HDFS system in parallel.

[0043] The following will take the HDFS system as an example for a detailed introduction.

[0044] After receiving the data backup request, each child node starts the data backup service used to ...

Embodiment 3

[0079] image 3 A schematic structural diagram of a data processing device according to Embodiment 3 of the present invention is shown. The device is applied to a data storage system including a master node and multiple sub-nodes, and the data processing devices in each sub-node run in parallel, such as image 3 As shown, the device includes: a data backup service starting module 300, a first reading module 301 and a backup module 302.

[0080] The data backup service starting module 300 is adapted to start the data backup service for connecting the child node and the intermediate storage system according to the data backup request, wherein the data backup service is pre-configured with a specified path for backing up data to the intermediate storage system.

[0081] The first reading module 301 is adapted to read the data distributed and stored in the child nodes in the form of a data table.

[0082] The backup module 302 is adapted to back up data in the form of data files...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing method, device, system, server and computer storage medium, which are applied to a data storage system including a main node and multiple sub-nodes. Wherein, the method is executed in parallel by each sub-node, including: according to the data backup request, starting the data backup service for connecting the sub-node and the intermediate storage system, wherein the data backup service is pre-configured with the data backup to the intermediate storage system Specify the path; read the data distributed and stored in the sub-nodes in the form of data tables, and according to the specified path, back up the data in the form of data files to the intermediate storage system through the data backup service. The data of the node is backed up to the intermediate storage system, thus improving the efficiency of data backup and saving time; the data is backed up to the intermediate storage system instead of discretely backing up to the sub-nodes locally, which is convenient for centralized management and improves security.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data processing method, device, system, server and computer storage medium. Background technique [0002] With the continuous development of information technology, distributed data storage systems appear, and distributed storage systems meet the needs of large-scale data storage. A distributed storage system usually includes different nodes, where the nodes may be nodes in different cabinets in the same computer room, and the computer rooms may be computer rooms in different locations. In practical applications, when the disk of the node fails or is damaged, the data stored in the node will be lost. In order to ensure data security, in a distributed storage system, data needs to be backed up. [0003] For a distributed data storage system that includes a master node and multiple sub-nodes, for example, the Greenplum data storage system generally only supports data ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F11/14H04L29/08
CPCG06F11/1448G06F11/1464G06F16/1844H04L67/1097
Inventor 张恒杨挺
Owner BEIJING QIHOO TECH CO LTD