Distributed data processing method and data center
A distributed data and data center technology, applied in the direction of electrical digital data processing, resource allocation, multi-programming devices, etc., can solve problems affecting the work efficiency of distributed data systems, achieve the goal of ensuring work efficiency and shortening processing time Effect
Active Publication Date: 2019-07-26
SHENZHEN LONGRISE SCI & TECH
View PDF5 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] The main purpose of the present invention is to provide a distributed data processing method and a data center, aiming to solve the technical problem that the existing distributed data system fragments the original data and seriously affects the working efficiency of the distributed data system
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0043] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.
[0044] The present invention provides a distributed data processing method. In the first embodiment of the distributed data processing method of the present invention, refer to figure 1 , the distributed data processing method includes:
[0045] Step S10, the data center analyzes the acquired raw data to obtain data parameters of the raw data;
[0046] The data center can include at least one server. The data center obtains raw data through the server. These raw data can be files or data packages uploaded by users through a network connection or a local connection. Data parameters include the data volume, data type, and version information of the original data. , Support parameters such as slicing processing, the data volume indicates the capacity of the original data, the data type includes image data type, text dat...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a distributed data processing method and a distributed data processing device. The distributed data processing method comprises the following steps of: acquiring acquired original data by a data center so as to acquire data parameters of the original data; dividing a task of slicing the original data into a plurality of sub-slicing tasks by the data center according to the data parameters and the data center properties; distributing external transferred sub-slicing tasks of a preset ratio in the sub-slicing tasks to data nodes except the data center in a distributed data system by the data center; implementing retained sub-slicing tasks which are not distributed in the sub-slicing tasks by the data center, storing corresponding first slicing data, and controlling the data nodes to implement the external transferred sub-slicing task and store corresponding second slicing data. By adopting the distributed data processing method, the situation that the original data are subjected to data slicing by only a single end of the data center is avoided, the time for data slicing processing on the original data is shortened, the consequence that normal operation of other courses of the distributed data processing system is delayed because of data slicing processing is avoided, and the working efficiency of the distributed data processing system is ensured.
Description
technical field [0001] The invention relates to the technical field of distributed data systems, in particular to a distributed data processing method and a data center. Background technique [0002] With the in-depth development of informatization, more and more data are generated by business systems in various industries. As the traditional centralized data system solution (centralized data system refers to the establishment of a specific access area for the data to be stored , access all data in the access area) store all data on the data center host for centralized data extraction, if the data center host fails, the entire system cannot store and extract data, that is, centralized data Compared with the distributed data system solution, the system solution has lower data stability, so the distributed data system came into being. [0003] The general process of the current distributed system for storing big data is: slice the original data in the data center, and after t...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50H04L29/08
CPCG06F9/4881G06F9/5027G06F2209/483G06F2209/5017H04L67/10
Inventor 张锐
Owner SHENZHEN LONGRISE SCI & TECH