Data synchronization method and device based on slice processing and electronic equipment
A data synchronization and data configuration technology, applied in database indexing, structured data retrieval, database distribution/replication, etc., can solve problems such as slow data synchronization, skewed data, and large differences in the number of data entries, and avoid data skewed, The effect of improving data synchronization efficiency and reducing data synchronization time
Pending Publication Date: 2022-08-09
上海淇玥信息技术有限公司
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
If the primary key uses different data types, the number of data pieces contained in each piece of data after splitting will vary greatly due to the difference between the value ranges of different data types, so the number of data pieces synchronized by each Task will also vary Very large, resulting in data skew, resulting in slow data synchronization
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0051] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings, although each exemplary embodiment can be embodied in various specific forms, it should not be construed that the invention is limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and fully convey the inventive concept to those skilled in the art.
[0052] The structures, performances, effects or other features described in a particular embodiment may be combined into one or more other embodiments in any suitable manner under the premise of conforming to the technical concept of the present invention.
[0053] In the description of specific embodiments, detailed descriptions of structures, performances, effects or other features are provided to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a data synchronization method and device based on slicing processing and electronic equipment. The method comprises the steps that a virtual slicing value is configured according to the data type of a data table primary key; dividing the data table primary key into a plurality of segmentation intervals based on the virtual segmentation value; performing slicing processing on the primary key of each segmentation interval to obtain a plurality of sub segmentation intervals; and performing data synchronization according to each sub-segmentation interval. According to the method, the virtual segmentation values are configured to distinguish the primary keys in different continuous storage ranges, and the primary keys in the same continuous storage range are divided into the same segmentation interval through the virtual segmentation values, so that each segmentation area only comprises the primary keys in the same continuous storage range, the primary key values in each segmentation interval are continuous and do not jump, and the storage efficiency is improved. And the primary key of each segmentation interval is sliced, so that the same number of data contained in each sub-segmentation interval in each segmentation interval after slicing is ensured as much as possible, data skew is avoided, the data synchronization time is shortened, and the data synchronization efficiency is improved.
Description
technical field [0001] The present invention relates to the technical field of data synchronization, and in particular, to a data synchronization method, apparatus, electronic device and computer-readable medium based on slice processing. Background technique [0002] Data synchronization is to synchronize data from different data sources to another data source through certain technologies and tools. In big data processing, the data in the relational database is generally synchronized to the data warehouse. At present, in relational databases, in order to meet the storage requirements of a large amount of business data, primary keys of different data types are used to configure data tables. [0003] In the data synchronization process, the data is usually divided into N (N>1 and a natural number) parts according to the primary key, and then N tasks are started to synchronize the N data in parallel. If the primary key adopts different data types, because of the differenc...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F16/27G06F16/22
CPCG06F16/27G06F16/2282
Inventor 刘坤元宋荣鑫黄建庭
Owner 上海淇玥信息技术有限公司