Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data synchronization system adaptive dynamic adjustment method and device

A data synchronization and dynamic adjustment technology, applied in database management systems, special data processing applications, structured data retrieval, etc., can solve problems such as inability to self-adjust, affect performance, lack of self-adaptive ability of worker threads, etc., to enhance the process The effect of adaptively adjusting processing power and improving overall performance

Active Publication Date: 2020-12-22
WUHAN DAMENG DATABASE
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] For each synchronization task, the first-level thread needs to allocate the total available memory of the synchronization process and the available memory of each functional component through calculation when starting. Each functional component cannot adaptively adjust the maximum memory required for this execution, and too much memory is allocated. Resources are wasted, and too little memory will directly affect performance. The repeated timing call execution of the process lacks self-learning ability and cannot effectively use memory; the synchronization process allocates the maximum amount of available memory during initialization, but it may not be able to fully utilize it during execution. Used, even if it is not used, the remaining memory cannot be used by other synchronous processes executed;
[0006] The number of first-level threads determines the maximum number of synchronization tasks that the data synchronization server can perform at the same time. Even if the computer hardware server has a large amount of idle CPU, memory and network, the task will not be executed when the first-level threads are exhausted. The newly added synchronization tasks in the queue lack the adaptive ability to dynamically increase or decrease worker threads according to the actual hardware and operating environment; when the data cache in the message queue of the upstream component of the functional component is large, it indicates the processing capability of the functional component Insufficient, the secondary processing thread of this functional component should be dynamically increased, the above solution does not have such a capability

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data synchronization system adaptive dynamic adjustment method and device
  • A data synchronization system adaptive dynamic adjustment method and device
  • A data synchronization system adaptive dynamic adjustment method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0054] An embodiment of the present invention provides a data synchronization system adaptive dynamic adjustment method,

[0055] For the convenience of description, the letters a-e are used to represent different cache amounts, among which:

[0056] The value of a indicates the maximum amount of cache that the functional component can apply to the system during this execution, which is recorded as the maximum available memory: maxCacheSize;

[0057] The b value indicates the maximum cache size that the functional component has obtained from the system according to the maximum cache size it may need during this execution, which is recorded as the current available cache: currentAvailableCacheSize;

[0058] The c value indicates the amount of cache actually used by the functional component during this execution, which is recorded as the currently used cache: currentUsingCacheSize;

[0059] The d value indicates the maximum cache amount actually used in the life cycle of the fu...

Embodiment 2

[0076] On the basis of Embodiment 1 above, the embodiment of the present invention provides a specific implementation method for dynamically adding primary threads and secondary threads during the synchronization process. It can be seen from Embodiment 1 that in the step 30, when the functional component adds data to its own message queue, if c≥b*present.1 is found, the functional component notifies its downstream component to add a secondary thread . In addition, the functional component can also notify its upstream component to add a secondary thread, specifically, when the functional component reads data from the message queue of its upstream component, if it finds that the message queue of the upstream component is empty or the message If the amount of cached data in the queue is less than the preset threshold, it means that the processing capability of the upstream component to read data or convert data is too low to keep up with the processing speed of the functional com...

Embodiment 3

[0088] On the basis of a data synchronization system adaptive dynamic adjustment method provided in Embodiment 1 and Embodiment 2, the present invention also provides a data synchronization system adaptive dynamic adjustment device that can be used to implement the above method, such as Image 6 Shown is a schematic diagram of the device architecture of the embodiment of the present invention. The data synchronization system adaptive dynamic adjustment device of this embodiment includes one or more processors 21 and memory 22 . in, Image 6 A processor 21 is taken as an example.

[0089] The processor 21 and the memory 22 may be connected via a bus or in other ways, Image 6 Take connection via bus as an example.

[0090] The memory 22, as a non-volatile computer-readable storage medium of a data synchronization system adaptive dynamic adjustment method, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, as in Embodim...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of data synchronization, in particular to a self-adaptive dynamic adjustment method and a device for a data synchronization system. The method comprises the following steps: determining a maximum available memory a that a functional component can apply to the system when synchronization is executed this time; according to the history log record, determining the maximum amount of delay memory b required for this execution, obtaining the corresponding memory from the system and starting the functional components; when the functional component appendsdata to its own message queue, updating the currently used cache c in real time. If c >= b*present.1 and b < a, it applies to the system for increasing memory. At that end of this execution, the maximum c value is log as the actual maximum usage cache for the next use. By increasing the maximum amount of slow memory for historical execution in the log, the functional component can determine the amount of cache usage required for this execution according to the log record, and adjust the memory adaptively at run time; at the same time, it can dynamically increase the first-level thread and thesecond-level thread according to the actual operation, and improve the performance of data synchronization.

Description

【Technical field】 [0001] The invention relates to the technical field of data synchronization, in particular to an adaptive dynamic adjustment method and device for a data synchronization system. 【Background technique】 [0002] In the data synchronization system, the synchronization process refers to the data synchronization task of data extraction, conversion, and loading composed of data reading components, data conversion components, and data loading components, including one or more data reading components, Zero or more data conversion components, one or more data loading components, and a synchronization process constitute a synchronization task. When a data synchronization task is executed, multiple execution threads are started. First, a first-level thread is started for the entire synchronization process to start the synchronization task described in the synchronization process, and second, one or more second-level threads are started for each functional component in...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2455G06F16/25G06F16/27
Inventor 付铨袁松彪梅纲
Owner WUHAN DAMENG DATABASE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com