Cluster data synchronization method and device based on big data, equipment and storage medium
A data synchronization and big data technology, applied in the field of data processing, can solve the problems of general stability and performance of massive data sets, lack of effective feedback on synchronization quality, lack of data synchronization solutions, etc., to improve efficiency and stability, avoid Hotspot writing problem, the effect of convenient storage and transmission
Pending Publication Date: 2020-02-14
CHINA PING AN LIFE INSURANCE CO LTD
View PDF8 Cites 9 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] In the existing technology, most of the synchronization solutions for big data are for the synchronization between clusters under the same system architecture, lack of data synchronization solutions between different types of clusters, and the data synchronization solutions are safe for cluster data The control is relatively loose, the computing engine used is slow, and the stability and performance of the synchronization of massive data sets are average, and there is no effective feedback on the quality of synchronization
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the application are only for the purpose of describing specific embodiments, and are not intended to limit the application.
[0048] It should be noted that the terms "comprising", "comprising" and "having" in the specification and claims of the present application and the above drawings and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally further includes For other steps or units inherent in these processes, methods, products or devices. In the claims, description and drawings of this application, re...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The embodiment of the invention discloses a cluster data synchronization method and device based on big data, equipment and a storage medium, and relates to the technical field of data processing. Themethod comprises the steps of receiving a data import instruction; in response to the data import instruction, obtaining to-be-synchronized data in the first cluster, and executing data conversion onthe to-be-synchronized data so as to convert the to-be-synchronized data into to-be-synchronized data matched with a storage structure in the second cluster; and establishing a plurality of parallelcommunication threads and a task monitoring table, synchronizing the converted to-be-synchronized data to the second cluster, and recording the synchronization state of the data in real time. According to the method, data synchronization between clusters which are not in the same system architecture can be realized. By means of data conversion, parallel thread setting and task monitoring table setting, synchronization efficiency and stability are improved, data such as task states, data loss and synchronization abnormity can be tracked in real time, and data states fed back during synchronization are obtained.
Description
technical field [0001] The embodiments of the present application relate to the technical field of data processing, in particular, a big data-based cluster data synchronization method, device, device, and storage medium. Background technique [0002] With the rapid development of the information age, the whole society is gradually entering the era of "digitalization". challenge. In this context, the calculation that consumes the most CPU has gradually shifted from improving the performance of the software itself to information processing, so that major manufacturers have to face great challenges—they need to start from TB (Terabyte, terabyte) level Even PB (Petabytes, Petabytes) level data can be mined to find useful information, and these massive data can be processed quickly and efficiently. Data storage is the cornerstone of data management, so how to store big data and migrate big data between different clusters is a problem worthy of attention and research. [0003] ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/22G06F16/23G06F16/182G06F16/178
CPCG06F16/27G06F16/2282G06F16/2255G06F16/23G06F16/182G06F16/178Y02D10/00
Inventor 饶刚
Owner CHINA PING AN LIFE INSURANCE CO LTD