Data synchronization method and system

Abstract

The invention provides a data synchronization method and system, the system comprises at least one client and at least one S3 object storage gateway, the to-be-synchronized data in the client is classified according to the size, and two to-be-synchronized file sets are obtained; performing aggregation processing on the files in each to-be-synchronized file set by utilizing an aggregation thread, and sending the aggregated files to a file aggregation vector buffer area; the files in the file aggregation vector buffer area are packaged into tasks to be transmitted through a scheduling thread, and the tasks to be transmitted are distributed to all CPU affinity transmission threads according to CPU core numbers; and the transmission thread uploads the files in the thread task sequence to a rear-end storage server in sequence according to the S3 object storage gateway transmission channel corresponding to the transmission thread. According to the method, the memory occupation of the data generated by the production equipment during the synchronization can be controllable, the load jitter is small, and the stability and the synchronization efficiency of the data synchronization of the low-performance equipment are effectively improved.

Description

Technical Field

[0001] This article belongs to the field of data transmission technology, and specifically relates to a data synchronization method and system. Background Art

[0002] In data-intensive fields such as manufacturing and hospitals, different device nodes will generate massive production data files during operation. The characteristics of these files are not only that the number is very large, but also that the file sizes sometimes vary greatly, ranging from a few bytes to tens of GB. If these device nodes are not in the core business, the performance of the equipment is often low in order to save investment. Considering security disaster recovery or business characteristics, these data files need to be synchronized to the back-end distributed storage system for backup at regular intervals. On low-performance production devices, the hardware performance of memory, CPU, disk, network, etc. is limited, but there is a need for synchronization of a large number of file data. Often the amount of data on a single device may reach millions, tens of millions, or even hundreds of millions of levels. In this scenario, it is extremely important, or even necessary, to ensure the stability and controllability of data synchronization of low-performance devices under heavy load scenarios. The hardware resources consumed by data synchronization, such as CPU, memory, network, etc., cannot affect production business, and the network topology structure for data synchronization to the back-end storage is a local area network or more cross-wide area networks.

[0003] Patent CN112162964B discloses "an adaptive data synchronization method, device, computer equipment and storage medium". The patent describes a data synchronization method, realizes multi-threaded parallel processing of multiple data directories, and adjusts the maximum value of the directory to be synchronized according to the CPU load parameter to improve the data synchronization efficiency. However, it does not involve the control and influence of device memory and network resources when synchronizing massive files in low-performance device scenarios and transmission across wide area networks. In the data synchronization business scenario, it is a typical IO-intensive business, which has limited impact on the CPU load, while the consumption of memory and bandwidth when synchronizing massive data files is often very large. When synchronizing data in a massive file heavy load scenario, the memory usage of the production equipment is controlled and the network resource occupancy is stable. At the same time, the CPU can be load balanced to eliminate load fluctuations and glitches. While completing data synchronization across the wide area network to the back-end storage, the data synchronization speed is maximized, the core business load intrusion is small, and the business in the production environment is safe and stable.

[0004] Therefore, existing data synchronization methods do not take into account the characteristics of low-performance devices. Either a large number of files are transmitted and synchronized in parallel, or individual files are transmitted and synchronized one by one. Either cross-WAN synchronization is not considered, or the peak consumption of device memory resources and bandwidth occupancy due to file size differences are not considered. This causes a certain amount of load jitter on hardware resources such as device node memory, CPU, bandwidth, etc., causing impact and uncontrollable production business. These problems seriously affect the stability and availability of the production environment, and in severe cases may even cause production business interruption.

[0005] Therefore, the prior art urgently needs a technical solution that can synchronize massive data of low-performance devices to back-end storage as quickly as possible under the premise of stability and controllability. Summary of the invention

[0006] In view of the above problems in the prior art, the purpose of this article is to provide a data synchronization method and system, which can realize fast, efficient, stable and controllable data synchronization of low-performance devices.

[0007] In order to solve the above technical problems, the specific technical solutions of this article are as follows:

[0008] On the one hand, this document provides a data synchronization method, which is applied to a data synchronization system including at least one client and at least one S3 object storage gateway, wherein the client and the S3 object storage gateway correspond to each other one by one, and the method includes:

[0009] According to the data synchronization directory in the client, classify the data to be synchronized in the client according to the size to obtain at least two sets of files to be synchronized;

[0010] Using the aggregation thread to aggregate the files in each set of files to be synchronized according to the fixed aggregation upper limit value, and sending the aggregated files to the file aggregation vector buffer;

[0011] The files in the file aggregation vector buffer are packaged into tasks to be transmitted by using a scheduling thread, and the tasks to be transmitted are distributed to the thread task sequence of each CPU affinity transmission thread according to the client CPU core number;

[0012] The transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the S3 object storage gateway transmission channel corresponding to it.

[0013] Furthermore, the set of files to be synchronized includes a first set of files to be synchronized and a second set of files to be synchronized, wherein the size of each file in the first set of files to be synchronized is smaller than a preset threshold, and the size of each file in the second set of files to be synchronized is not smaller than the preset threshold.

[0014] Furthermore, when the aggregate file includes multiple files, the scheduling thread generates a corresponding number of transmission tasks according to the number of files in the file aggregation vector buffer, and schedules them to the transmission thread groups bound to the CPU in sequence according to the CPU core number.

[0015] Furthermore, when there is only one file in the aggregate file, the scheduling thread packages the files in the file aggregation vector buffer into a transmission task and schedules it to the transmission thread with the smallest load, so that the transmission thread transmits the file according to the fragment transmission strategy, and the size of the file is greater than the fixed value of the aggregation upper limit.

[0016] Furthermore, the transmission thread with the least load is determined by the following steps:

[0017] Get the queue depth of each transmission thread and the number of tasks currently to be transmitted;

[0018] According to the queue depth of each transmission thread and the number of tasks to be transmitted, the load value of each transmission thread is calculated to determine the transmission thread with the smallest load.

[0019] Furthermore, the transmission thread transmits the file according to the fragment transmission strategy, including:

[0020] According to the fixed value of the upper limit of aggregation, the transmission thread performs sharding processing on the file to obtain sharded data blocks with a transmission time sequence, and the transmission thread sequentially transmits data according to the corresponding S3 object storage gateway transmission channel;

[0021] During the data transmission process of the transmission thread, the slice prefetch thread is used to asynchronously and parallelly complete the prefetching work of the next slice data block of the slice data block currently being transmitted, and the slice data block is prefetched into the prefetch data buffer.

[0022] Furthermore, the method further comprises:

[0023] The transmission thread performs marking processing on each shard data block. When all the shard data blocks are transmitted, the transmission thread notifies the S3 object storage server that the transmission of the file has been completed.

[0024] Furthermore, the method further comprises:

[0025] A data synchronization plan is configured, wherein the data synchronization plan includes at least a data file directory to be synchronized on the client, configuration information of a backend storage server, and a unique data synchronization task name.

[0026] On the other hand, the present invention also provides a data synchronization system, the system comprising at least one client and at least one S3 object storage gateway, the client and the S3 object storage gateway correspond to each other, and each of the S3 object storage gateways communicates with a backend storage server via a local area network;

[0027] Different clients are distributed in different network topologies, and the clients communicate with their corresponding S3 object storage gateways through a wide area network;

[0028] Each client is configured with a data synchronization service to synchronize internal data to the backend storage server. The data synchronization service performs data synchronization through the following steps:

[0029] According to the data synchronization directory in the client, classify the data to be synchronized in the client according to the size to obtain at least two sets of files to be synchronized;

[0030] Using the aggregation thread to aggregate the files in each set of files to be synchronized according to the fixed aggregation upper limit value, and sending the aggregated files to the file aggregation vector buffer;

[0031] The files in the file aggregation vector buffer are packaged into tasks to be transmitted by using a scheduling thread, and the tasks to be transmitted are distributed to the thread task sequence of each CPU affinity transmission thread according to the client CPU core number;

[0032] The transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the S3 object storage gateway transmission channel corresponding to it.

[0033] Optionally, the client is a low-performance production device.

[0034] Adopting the above technical scheme, a data synchronization method and system described in this article, the system includes a data synchronization system of at least one client and at least one S3 object storage gateway, the client and the S3 object storage gateway correspond one to one, and then according to the data synchronization directory in the client, the data to be synchronized in the client is classified according to size to obtain at least two sets of files to be synchronized; the aggregation thread is used to aggregate the files in each set of files to be synchronized according to the fixed value of the aggregation upper limit, and the aggregated files are sent to the file aggregation vector buffer; the scheduling thread is used to package the files in the file aggregation vector buffer as tasks to be transmitted, and the tasks to be transmitted are distributed to the thread task sequence of each CPU affinity transmission thread according to the client CPU core number; the transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the corresponding S3 object storage gateway transmission channel. The technical scheme disclosed in this article can ensure that the memory occupancy of the data generated by the production equipment during the cross-wide area network synchronization is controllable, the load jitter is small, and the network resource occupancy is small, which effectively improves the stability and synchronization efficiency of the data synchronization of low-performance devices to the backend object storage.

[0035] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic diagram of a data synchronization system provided by an embodiment of the present invention is shown;

[0038] Figure 2 A schematic diagram showing the steps of the data synchronization method provided in the embodiment of this article is shown;

[0039] Figure 3 A schematic diagram of file aggregation scheduling in an embodiment of this invention is shown;

[0040] Figure 4 A schematic diagram of the framework of a computer device provided in the embodiments of this document is shown. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0043] In the prior art, when synchronizing data generated by low-performance devices, especially a large number of small files, the characteristics of the devices are not taken into account. Either a large number of files are transmitted and synchronized in parallel, or a single file is transmitted and synchronized one by one. Either synchronization across the wide area network is not taken into account, or the peak consumption of device memory resources and bandwidth occupancy due to file size differences are not taken into account. This causes a certain amount of load jitter on hardware resources such as device node memory, CPU, bandwidth, etc., causing impact and uncontrollability on production business. These problems seriously affect the stability and availability of the production environment, and in severe cases may even cause production business interruption.

[0044] In order to solve the above problems, the embodiments of this document provide a data synchronization method, which is suitable for a data synchronization system and can improve the synchronization efficiency of low-performance devices. Figure 1 This is a framework diagram of a data synchronization system provided in an embodiment of this invention, wherein the system comprises at least one client and at least one S3 object storage gateway, wherein the client and the S3 object storage gateway correspond one to one, and each of the S3 object storage gateways communicates with a backend storage server via a local area network; different clients are distributed in different network topologies, and the clients communicate with their corresponding S3 object storage gateways via a wide area network.

[0045] Exemplarily, the data synchronization system includes multiple backup clients to be synchronized and three S3 object gateway servers placed on the public network. The hardware of each gateway server includes CPU, memory, network interface, multiple storage servers placed in the same local area network as the S3 object gateway server, and multiple hard disks for storing data. Multiple storage servers can be distributed. The software running on the storage server includes operating system and storage management software. Different S3 object storage gateway servers and storage service servers communicate through the internal network of the data center, and the backup client and S3 object gateway communicate through the wide area network.

[0046] Among them, the backup client can be the generation node of the data source, which is generally distributed in different places according to the actual production environment, such as hospital offices, hospital data centers, factory office areas, production workshops, etc. The important data generated by these nodes have synchronization requirements as needed, and the data file directory to be synchronized needs to be specified during data synchronization.

[0047] Optionally, the client can be a low-performance device, which can be understood as a device with insufficient hardware performance, such as a device with low parameters such as network, disk, memory, etc. The performance of the production equipment is not high, the size of the generated data files varies greatly, and the number of data files is very large; the devices are scattered in multiple network topologies, and may be in the same data center, the same local area network or wide area network; the backend S3 object storage is located in the wide area network; the data synchronization service is deployed on the production device node, the data synchronization source and the backend object storage cross the wide area network, and each data synchronization node will be assigned a unique data synchronization identification name, the data file directory to be synchronized, and specify the service address of the backend object storage.

[0048] The S3 object storage gateway is generally deployed on the public network (i.e., wide area network) and forms an internal network with the backend storage system. After the backup client is synchronized to the S3 object storage through the data synchronization service, it will be forwarded to the backend storage system.

[0049] The data synchronization service runs on the backup client, executes the data synchronization plan according to certain configuration rules, and synchronizes data to the S3 object storage gateway through the S3 standard protocol.

[0050] Each client is configured with a data synchronization service to synchronize internal data to the backend storage server. The data synchronization service performs data synchronization through the following steps:

[0051] According to the data synchronization directory in the client, classify the data to be synchronized in the client according to the size to obtain at least two sets of files to be synchronized;

[0052] Using the aggregation thread to aggregate the files in each set of files to be synchronized according to the fixed aggregation upper limit value, and sending the aggregated files to the file aggregation vector buffer;

[0053] The files in the file aggregation vector buffer are packaged into tasks to be transmitted by using a scheduling thread, and the tasks to be transmitted are distributed to the thread task sequence of each CPU affinity transmission thread according to the client CPU core number;

[0054] The transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the S3 object storage gateway transmission channel corresponding to it.

[0055] Through the above-mentioned data synchronization system, in conjunction with the data synchronization execution method, it can be ensured that during the synchronization of massive data of production environment equipment across the wide area network, the device memory occupancy is controllable, the load jitter is small, the network resource occupancy is small, and the impact on core business is small. It can effectively improve the stability and synchronization efficiency of data synchronization from low-performance devices with insufficient hardware performance such as network disk memory to the back-end object storage, thereby ensuring the stability of production business in the production environment and the continuity of business.

[0056] like Figure 2 As shown, it is a schematic diagram of the steps of the data synchronization method provided by the data synchronization system. This specification provides the method operation steps described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in the order or in parallel according to the method shown in the embodiment or the drawings. Specifically, Figure 2 As shown, the method may include:

[0057] S201: Classifying the data to be synchronized in the client according to the size according to the data synchronization directory in the client to obtain at least two sets of files to be synchronized;

[0058] S202: using an aggregation thread to aggregate files in each set of files to be synchronized according to a fixed aggregation upper limit, and sending the aggregated files to a file aggregation vector buffer;

[0059] S203: using a scheduling thread to package the files in the file aggregation vector buffer into tasks to be transmitted, and distributing the tasks to be transmitted to thread task sequences of each CPU affinity transmission thread according to the client CPU core number;

[0060] S204: The transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the corresponding S3 object storage gateway transmission channel.

[0061] It can be understood that this manual configures aggregation threads, scheduling threads, and transmission threads on the client, classifies and aggregates files to be synchronized in batches, and binds the transmission threads with CPU affinity to the CPU core to ensure that the data is scheduled to be processed by a better transmission thread during transmission. This can control the memory usage, reduce network resources, and effectively improve the stability and synchronization efficiency of data synchronization to the backend object storage under low-performance devices such as network disk memory with insufficient hardware performance. Figure 3 As shown, this is a schematic diagram of file aggregation scheduling in this article.

[0062] The file aggregation vector buffer may be a memory area of ​​a priority queue, and may be configured in advance according to the performance storage of different clients.

[0063] Furthermore, the set of files to be synchronized includes a first set of files to be synchronized and a second set of files to be synchronized, wherein the size of each file in the first set of files to be synchronized is smaller than a preset threshold, and the size of each file in the second set of files to be synchronized is not smaller than the preset threshold.

[0064] For example, the preset threshold may be 64M, that is, the file size of 64M is used as the classification threshold, and files smaller than 64M enter the first set of files to be synchronized as small files; files not smaller than 64M enter the second set of files to be synchronized as large files, thereby obtaining two types of file sets. By classifying the files, the efficiency of subsequent centralized processing of the files is facilitated, and small files can be processed in a centralized manner, thereby improving the efficiency of small file transmission.

[0065] Furthermore, when the aggregate file includes multiple files, the scheduling thread generates a corresponding number of transmission tasks according to the number of files in the file aggregation vector buffer, and schedules them to the transmission thread groups bound to the CPU in sequence according to the CPU core number.

[0066] It can be understood that this process is mainly to aggregate the files in the first set of files to be synchronized to obtain an aggregated file. Since the first set of files to be synchronized contains small files, the aggregated file obtained by aggregation can be multiple small files. In this way, when transmitting, the corresponding number of transmission threads can be allocated according to the number of small files in the aggregated file in the file aggregation vector buffer. At this time, the transmission thread is a transmission thread bound to the CPU affinity in advance, and each transmission thread will establish an independent S3 object storage transmission channel, so as to realize the synchronous transmission of multiple small files, and realize fast and efficient data transmission while ensuring a small memory usage.

[0067] When there is only one file in the aggregate file, the scheduling thread packages the files in the file aggregation vector buffer into a transmission task and schedules it to the transmission thread with the smallest load, so that the transmission thread transmits the file according to the fragment transmission strategy, and the size of the file is greater than the fixed upper limit of the aggregation.

[0068] It can be understood that, for large files, the embodiments of the present specification can schedule them to be processed by the transmission thread with the smallest load, which can achieve smaller load jitter and ensure the efficiency and reliability of large file transmission.

[0069] The transmission thread with the least load is determined by the following steps:

[0070] Get the queue depth of each transmission thread and the number of tasks currently to be transmitted;

[0071] According to the queue depth of each transmission thread and the number of tasks to be transmitted, the load value of each transmission thread is calculated to determine the transmission thread with the smallest load.

[0072] Exemplarily, assuming that the number of tasks on a certain transmission thread T is N, the size of data to be transmitted by each task is K1~KN, and the queue depth of the transmission thread is Q, then the load calculation formula for each transmission thread is TL=(K1+K2+...+Ki+...+KN)*0.8+Q*0.2. The lowest transmission load on the CPU corresponding to the transmission thread is confirmed through TL, all CPU cores in the client are obtained, the current load of each CPU core is queried, the CPU with the smallest load is found, and the transmission task corresponding to the large file is assigned to the transmission thread on the CPU core with the smallest load.

[0073] When transmitting large files, since the large files themselves have a large amount of data, a long transmission time is required during the transmission. In addition, combined with the real-time change of the CPU core load during the transmission process, data loss may occur during the transmission process. In order to ensure the reliability and stability of the large file data transmission, this article can adopt a fragment transmission strategy for transmission. Optionally, the transmission thread transmits the file according to the fragment transmission strategy, including:

[0074] According to the fixed value of the upper limit of aggregation, the transmission thread performs sharding processing on the file to obtain sharded data blocks with a transmission time sequence, and the transmission thread sequentially transmits data according to the corresponding S3 object storage gateway transmission channel;

[0075] During the data transmission process of the transmission thread, the slice prefetch thread is used to asynchronously and parallelly complete the prefetching work of the next slice data block of the slice data block currently being transmitted, and the slice data block is prefetched into the prefetch data buffer.

[0076] It can be understood that the shard transmission strategy can be used to pre-process large files into blocks, and a large file composed of multiple small shard data blocks can be obtained. In this way, when the transmission thread transmits a large file, it actually transmits each shard data block in sequence, reducing the amount of data transmitted by the transmission thread each time. For example, the fixed value of the aggregation upper limit can be 64M, so that the size of each shard data block does not exceed 64M, reducing the load on the CPU core during each transmission and ensuring the stability of the transmission.

[0077] The shard prefetch thread can reduce the preparation time for extracting adjacent shard data blocks, fully utilize the working performance of the CPU core, reduce load jitter, and improve the transmission efficiency of the entire large file transmission process.

[0078] Of course, for large files, multiple shard data blocks are transmitted in sequence. In order to avoid omission of shard data blocks, the shard data blocks can be marked in advance to ensure that the back-end storage server can obtain and respond to the complete large file in time. Optionally, the method also includes:

[0079] The transmission thread performs a marking process on each shard data block. When all the shard data blocks are transmitted, the transmission thread notifies the S3 object storage server that the file transmission has been completed. For example, the transmission sequence of the shard data blocks can be marked. When the transmission is completed according to the transmission sequence, the transmission thread will transmit a notification that the current large file transmission is completed, so that the back-end storage server can integrate the received shard data blocks to obtain a complete large file.

[0080] In the embodiment of this specification, the method further includes:

[0081] Configure a data synchronization plan, which at least includes the client's data file directory to be synchronized, the configuration information of the backend storage server, and a unique data synchronization task name. By configuring the data synchronization rules, the data is automatically saved to the corresponding data file target when it is generated, which can realize automatic data synchronization and realize seamless and efficient synchronization.

[0082] In another embodiment of the present specification, a data synchronization method is also provided. Figure 4 As shown, the following process may be included:

[0083] 1. Configure the data file directory to be synchronized, the configuration information of the remote object storage, the unique data synchronization task name, and the data synchronization plan on the backup client.

[0084] 2. The data synchronization service performs a system self-check, including checks on the network, CPU, disk, data synchronization directory permissions, and S3 protocol object storage read and write functions. If the system self-check passes, data synchronization is started according to the data synchronization task plan.

[0085] 3. If the system self-test fails, the service will restart at a fixed interval and then perform self-test again, and then go to step 2. If the self-test passes, then go to step 4.

[0086] 4. The data synchronization service builds data synchronization tasks according to the data synchronization plan, and these tasks may change at any time with different data synchronization strategies. If the data synchronization plan changes, the existing data synchronization tasks will be canceled, and the data synchronization tasks being executed will need to wait until they are completed. The next data synchronization cycle will definitely not be executed again. Newly generated data synchronization tasks will be generated and scheduled into the task queue. If the backup client cancels the data synchronization plan, all subsequent data synchronization tasks will be canceled after the ongoing data synchronization tasks are completed.

[0087] 5. When data synchronization is in progress, multiple business threads will be running, including aggregation thread, scheduling thread, transmission thread, and shard prefetch thread. At the same time, there will be two buffers for asynchronous pipeline collaboration, namely the file object aggregation vector buffer and the shard prefetch buffer.

[0088] 6. When data synchronization is in progress, the following steps will be followed to complete the data synchronization:

[0089] a) Divide the files in the specified data synchronization directory into two types according to their size: small files (for example, less than 64MB) and large files (for example, 64M and above);

[0090] b) The aggregation thread aggregates one or more files into the file aggregation vector buffer according to the fixed value of the file size aggregation upper limit (here 64MB);

[0091] c) The scheduling thread packages the files in the file aggregation vector buffer into tasks to be transmitted. The transmission tasks here include not only the content of the file itself but also the context information required for file transmission, and distribute them to each CPU affinity transmission thread in sequence according to the CPU core number;

[0092] d) Each transmission thread will establish an independent S3 object storage transmission channel, and the transmission thread will notify b) that all tasks scheduled into the thread have completed transmission;

[0093] e) After the transmission thread completes the task assigned to it this time, it waits for the scheduling thread to package the files in the object aggregation vector buffer as tasks and deliver them to its own thread task queue;

[0094] f) If the files in the file aggregation vector buffer are a large number of small files, the scheduling thread will generate a corresponding number of transmission tasks according to the number of files in the file aggregation vector buffer;

[0095] g) The scheduling thread schedules the transmission tasks generated by f) to the transmission thread groups d) bound to the CPU in sequence according to the CPU core number;

[0096] h) The transmission threads upload the files in the transmission task queue to the backend object storage in sequence through their respective S3 object storage connections, and each transmission channel in the transmission thread group performs parallel transmission;

[0097] i) If the file in the file aggregation vector buffer is a large file larger than 64M, the scheduling thread will package the files in the file aggregation vector buffer (at this time, there is only one file in the file aggregation buffer, which is generally a large file) into a transmission task and assign it to the thread with the smallest load on the transmission thread;

[0098] j) Confirm the transmission binding with the smallest load in step i). The load is determined as follows: Assuming that the number of tasks on a transmission thread T is N, the data size of the tasks is K1-KN, and the queue depth of the transmission thread is Q, the load calculation formula for each transmission thread is TL=(K1+K2+...+Ki+...+KN)*0.8+Q*0.2. Confirm the lowest transmission load on the transmission thread CPU through TL, and assign the transmission task in step i) to the transmission thread on the CPU core with the smallest load;

[0099] k) The transmission thread divides the large file into 64M pieces, and the data read each time does not exceed 64MB;

[0100] The formed shard data is transmitted in the S3 object storage transmission channel established by itself;

[0101] l) During the k)-step transmission process, the shard prefetch thread asynchronously and parallelly completes the prefetching of the next 64MB data block of the file being transmitted, and completes the prefetching of the shard data into the prefetch data buffer;

[0102] m)l) During the transmission process, each shard data of a large file will be marked when it is uploaded through the S3 object storage transmission channel. After the last shard is transmitted, the S3 object storage gateway will be notified that the transmission of a complete large file is completed.

[0103] 7. All files in the data synchronization directory are aggregated into the object aggregation vector buffer by the aggregation thread, and the data synchronization is completed after the transmission thread synchronizes all files to the backend object storage, and the audit log is recorded.

[0104] The purpose of the embodiments of this specification is to provide a data synchronization method, that is, a stable and fast synchronization method for massive data of low-performance devices across a wide area network, which is used to complete data synchronization from machine nodes in a production environment to back-end wide area network object storage. The method divides the data files generated by the production equipment into two types according to the file size: small files (less than 64MB) and large files (64MB and above). During data synchronization, an aggregation thread is used to aggregate one or more files into a file aggregation buffer according to a fixed size (64MB) upper limit, and the scheduling thread is notified to package the files in the buffer into transmission tasks and distribute them to the fixed transmission threads bound to each CPU core. Each transmission thread independently establishes an S3 object storage transmission channel. After the transmission thread completes the transmission of all the transmission tasks packaged this time, it waits for the scheduling thread to notify again to package the files in the file aggregation buffer into transmission tasks. The aggregation thread aggregates one or more files into the buffer again according to the 64M upper limit. If the files aggregated at a certain time are a large number of small files, the scheduling thread will generate a corresponding number of transmission tasks according to the number of files in the buffer, and then dispatch them to each transmission thread bound to the CPU in sequence according to the CPU core number. These transmission threads upload the entire independent file in the transmission task to the S3 object storage backend at one time through their respective S3 object storage connections, and each transmission channel completes parallel transmission. If the aggregated file is a large file larger than 64M, only one transmission task will be packaged from the buffer and assigned to the transmission thread on the CPU core with the smallest load on the CPU core. At this time, the transmission thread will fragment the large file into 64M pieces, and the data read each time will not exceed 64MB. The fragmented data will be transmitted in its own S3 object storage transmission channel. At the same time, the prefetch thread will asynchronously complete the prefetching of the next 64MB data block of the large file being transmitted during the transmission process of the transmission thread, and prefetch it into the user-state buffer in advance to prepare for the next data transmission. Each fragmented data will be marked, and after the last fragment is transmitted, the S3 object storage gateway will be notified to complete the data synchronization of the large file after the fragment transmission. This technology aggregates and transmits small files of a fixed size (64MB) in parallel according to a data synchronization, and asynchronously pre-fetches pipeline transmission of fixed-size fragments of large files. This can ensure that the memory occupancy of equipment in the production environment is controllable during the synchronization of massive data across the WAN, with small load jitter, small network resource usage, and little impact on core business. It can effectively improve the stability and efficiency of data synchronization from low-performance devices with insufficient hardware performance, such as network disk memory, to the back-end object storage, and ensure the stability of production business in the production environment and the continuity of business.

[0105] This embodiment provides a computer device, whose internal structure diagram can be shown as follows: Figure 4As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection.

[0106] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0108] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0109] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0111] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0116] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A data synchronization method, characterized in that: Applied to a data synchronization system including at least one client and at least one S3 object storage gateway, wherein the client and the S3 object storage gateway correspond one to one, the method includes: According to the data synchronization directory in the client, classify the data to be synchronized in the client according to the size to obtain at least two sets of files to be synchronized; Using the aggregation thread to aggregate the files in each set of files to be synchronized according to the fixed aggregation upper limit value, and sending the aggregated files to the file aggregation vector buffer; The files in the file aggregation vector buffer are packaged into tasks to be transmitted by using a scheduling thread, and the tasks to be transmitted are distributed to the thread task sequence of each CPU affinity transmission thread according to the client CPU core number; The transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the S3 object storage gateway transmission channel corresponding to it.

2. The method according to claim 1, characterized in that The to-be-synchronized file set includes a first to-be-synchronized file set and a second to-be-synchronized file set, wherein the size of each file in the first to-be-synchronized file set is smaller than a preset threshold, and the size of each file in the second to-be-synchronized file set is not smaller than the preset threshold.

3. The method according to claim 2, characterized in that When the aggregate file includes multiple files, the scheduling thread generates a corresponding number of transmission tasks according to the number of files in the file aggregation vector buffer, and schedules them to the transmission thread groups bound to the CPU in sequence according to the CPU core number.

4. The method according to claim 2, characterized in that: When there is only one file in the aggregate file, the scheduling thread packages the files in the file aggregation vector buffer into a transmission task and schedules it to the transmission thread with the smallest load, so that the transmission thread transmits the file according to the fragment transmission strategy, and the size of the file is greater than the fixed upper limit of the aggregation.

5. The method according to claim 4, characterized in that The transmission thread with the least load is determined by the following steps: Get the queue depth of each transmission thread and the number of tasks currently to be transmitted; According to the queue depth of each transmission thread and the number of tasks to be transmitted, the load value of each transmission thread is calculated to determine the transmission thread with the smallest load.

6. The method according to claim 4, characterized in that The transmission thread transmits the file according to the fragment transmission strategy, including: According to the fixed value of the aggregation upper limit, the transmission thread performs sharding processing on the file to obtain sharded data blocks with a transmission time sequence, and the transmission thread sequentially transmits data according to the corresponding S3 object storage gateway transmission channel; During the data transmission process of the transmission thread, the slice prefetch thread is used to asynchronously and parallelly complete the prefetching work of the next slice data block of the slice data block currently being transmitted, and the slice data block is prefetched into the prefetch data buffer.

7. The method according to claim 6, characterized in that The method further comprises: The transmission thread performs marking processing on each shard data block. When all the shard data blocks are transmitted, the transmission thread notifies the S3 object storage server that the transmission of the file has been completed.

8. The method according to claim 1, characterized in that The method further comprises: A data synchronization plan is configured, wherein the data synchronization plan includes at least a data file directory to be synchronized on the client, configuration information of a backend storage server, and a unique data synchronization task name.

9. A data synchronization system, characterized in that: The system includes at least one client and at least one S3 object storage gateway, the client and the S3 object storage gateway correspond to each other, and each of the S3 object storage gateways communicates with a backend storage server through a local area network; Different clients are distributed in different network topologies, and the clients communicate with their corresponding S3 object storage gateways through a wide area network; Each client is configured with a data synchronization service to synchronize internal data to the backend storage server. The data synchronization service performs data synchronization through the following steps: According to the data synchronization directory in the client, classify the data to be synchronized in the client according to the size to obtain at least two sets of files to be synchronized; Using the aggregation thread to aggregate the files in each set of files to be synchronized according to the fixed aggregation upper limit value, and sending the aggregated files to the file aggregation vector buffer; The files in the file aggregation vector buffer are packaged into tasks to be transmitted by using a scheduling thread, and the tasks to be transmitted are distributed to the thread task sequence of each CPU affinity transmission thread according to the client CPU core number; The transmission thread uploads the files in the thread task sequence to the backend storage server in sequence according to the S3 object storage gateway transmission channel corresponding to it.

10. The system according to claim 9, characterized in that The client is a low-performance production device.

Images