Unlock instant, AI-driven research and patent intelligence for your innovation.

A data parallel processing method

A parallel processing and data processing technology, applied in the field of data processing, can solve the problems of low utilization rate, inability to use data, difficult to adapt to high intensity, etc., to achieve balance between performance and storage capacity, shorten merger time, and improve data processing efficiency. Effect

Active Publication Date: 2019-08-02
上海融甸信息科技有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, there is a conflict between data processing and data use, making it impossible to load data during data use, and to use data during data loading
The utilization rate of the system is not high, and it is difficult to adapt to high-intensity and long-term analysis
However, the usual method of two mirrored data brings difficulties to the storage of large amounts of data (the data environment used reaches a capacity of more than 10T) and synchronization (the commonly used database-level log synchronization technology is difficult to adopt)

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data parallel processing method
  • A data parallel processing method
  • A data parallel processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] The present invention will be described in detail below in conjunction with specific embodiments. This embodiment is carried out on the premise of the technical solution of the present invention, and detailed implementation and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.

[0020] This embodiment provides a data parallel processing method, which includes a data processing stage, a data use stage, and a data merging stage that are independent of each other, the data processing stage and the data use stage are parallel, and the data use stage and the data merging stage Rebuke. Data processing, data merging, and data usage operate independently, and data consistency is achieved through state control.

[0021] The data merging stage is to replace and add operations to the data history area. The data history area is divided into a current partition and a history partition. The current pa...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a parallel data processing method. The method comprises a data processing stage, a data using stage and a data combination stage, which are mutually independent; the data processing stage is parallel to the data using stage; the data using stage is exclusive with the data combination stage; in the data combination stage, a data history region is subjected to replacement and addition operations; the data history region is divided into a current region and a history region; the current region is used for storing data without end time; and the history region is used for storing data with end time. Compared to the prior art, the parallel data processing method has the advantages that parallel data using and loading are realized, the data synchronicity is improved and the like.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a data parallel processing method. Background technique [0002] Data use requires that the data be consistent at the start time during the data use period. After the data starts to be used, the data cannot be added, deleted or modified. At the same time, table locks are often used on the timetable to improve the efficiency of data loading. Therefore, there is a conflict between the data processing process and the data use, so that the data cannot be loaded during the data use, and the data cannot be used during the data load. The utilization rate of the system is not high, and it is difficult to adapt to high-intensity and long-term analysis. However, the usual method of two mirrored data brings difficulties to the storage of large amounts of data (the data environment used reaches a capacity of more than 10T) and synchronization (the commonly used database-level log synchronizat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/21
CPCG06F16/2282G06F16/25
Inventor 吴卫军范磊
Owner 上海融甸信息科技有限公司