Data synchronization method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a data synchronization method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of computers, the method is applied to a source cluster, and the method specifically comprises the following steps: reading at least two business operation records pre-solidified on a disk, and constructing a first queue; business operation records in the first queue are checked, target business operation records meeting preset requirements are moved to a second queue, and the target business operation records are at least one business operation record in the at least two business operation records; and sending the service operation record in the second queue to a target cluster. According to the embodiment of the invention, the service performance loss in cross-cluster synchronization can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, device, electronic device, storage medium, and program product. Background Art

[0002] At present, cross-cluster synchronization technologies mainly include LAN cross-cluster synchronization and WAN cross-cluster synchronization. Among them, LAN cross-cluster synchronization is to synchronize data between different clusters within the same LAN, specifically using a synchronous waiting method. Only when both the source cluster and the destination cluster have successfully processed, the source cluster responds and sends a success message to the client. WAN cross-cluster synchronization obtains the metadata information to be synchronized through the interaction between the source cluster client and the metadata management module of this cluster. The client sends a cross-cluster network synchronization message to the destination cluster client so that the destination cluster client notifies the destination cluster metadata management module to replay the metadata operation. However, if the source cluster and the destination cluster are far apart, the use of LAN cross-cluster synchronization technology will cause the network to consume a long time, thereby affecting business performance. The use of WAN cross-cluster synchronization technology requires multiple interactions between clusters, and the complicated interaction process leads to low synchronization efficiency, which in turn affects business performance. It can be seen that the cross-cluster synchronization technology in the relevant technology has the problem of large business performance loss. Summary of the Invention

[0003] Embodiments of the present application provide a data synchronization method, apparatus, electronic device, storage medium, and program product, which can reduce service performance loss in cross-cluster synchronization.

[0004] In a first aspect, an embodiment of the present application provides a data synchronization method, applied to a source cluster, the method comprising:

[0005] Read at least two business operation records pre-solidified on the disk to build a first queue;

[0006] Checking the business operation records in the first queue and moving target business operation records that meet preset requirements to the second queue, the target business operation record being at least one business operation record of the at least two business operation records;

[0007] The business operation records in the second queue are sent to the target cluster.

[0008] Optionally, before reading the at least two business operation records pre-solidified on the disk, the method further includes:

[0009] Determine the CPU core number on which each of the business operation records is running based on the name, operation record type, and number of CPU cores of each of the business operation records;

[0010] Binding each of the business operation records to a corresponding CPU core number;

[0011] Determining the disk range corresponding to each business operation record according to the CPU core number corresponding to each business operation record;

[0012] According to the serial number of each business operation record, the serial number corresponding to each business operation record is written into the corresponding disk area in sequence.

[0013] Optionally, reading the at least two business operation records recorded on the disk to construct a first queue includes:

[0014] Concurrently scanning a specified interval in the disk through multiple channels to obtain channel queues corresponding to multiple scanning threads; wherein the multiple scanning threads are threads formed by the source cluster scanning the specified interval through the multiple channels, and the channel queue includes at least one business operation record located in the specified interval;

[0015] Obtaining an operation type or a corresponding CPU core number of a first business operation record in each channel queue, where the first business operation record is the business operation record arranged at the head of the channel queue, and the sequence number of the first business operation record is the smallest sequence number among the sequence numbers of multiple business operation records in the channel queue where the first business operation record is located;

[0016] Based on the operation type of the first business operation record in the channel queues corresponding to each of the multiple scanning threads or the CPU core number corresponding to the first business operation record, a first operation is performed to obtain the first queue.

[0017] Optionally, the first operation includes at least one of the following:

[0018] When the operation type of the first business operation record in each channel queue is consistent, the business operation record in each channel queue is sequentially added to a preset queue to obtain the first queue;

[0019] In the case that there is at least one channel queue in which the operation type of the first business operation record is the target operation type, the first business operation record with the operation type of the target operation type is added to the preset queue to obtain the first queue.

[0020] Optionally, checking the business operation records in the first queue and moving target business operation records that meet preset requirements to the second queue includes:

[0021] assigning a corresponding weight to each business operation record in the first queue according to an operation type of each business operation record in the first queue;

[0022] If a second business operation record is arranged after a third business operation record in the first queue and a weight of the second business operation record is less than or equal to a weight of the third business operation record, determining that the state of the second business operation record is a first state; wherein the second business operation record and the third business operation record are any two business operation records in the first queue, and processing of the second business operation record is suspended in the first state;

[0023] If the fourth business operation record is arranged after the third business operation record in the first queue and the weight of the fourth business operation record is greater than the weight of the third business operation record, determining that the status of the fourth business operation record is the second status;

[0024] If the fourth business operation record is in the second state, determining, based on an operation type of the fourth business operation record and a pre-acquired hash value, whether the fourth business operation record has been classified into a third queue, the third queue being used to store business operation records related to a second operation, the second operation including at least one of creation, deletion, rename, hard link, and soft link;

[0025] If the fourth business operation record is not included in the third queue, the fourth business operation record is moved to the second queue.

[0026] Optionally, the method further includes:

[0027] If the fourth business operation record has been included in the third queue, determining whether a fifth business operation record exists in the third queue based on a hash value of the fourth business operation record; wherein the name of the fifth business operation record is the same as the name of the fourth business operation record;

[0028] If the fifth business operation record exists in the third queue and synchronization processing of the fifth business operation record has not been completed, move the fourth business operation record to the second queue;

[0029] When the fifth business operation record does not exist in the third queue, the fourth business operation record is moved to the second queue.

[0030] Optionally, sending the business operation record in the second queue to the target cluster includes:

[0031] Based on the multi-channel network constructed between the source cluster and the target cluster, concurrently sending the business operation records in the second queue to the target cluster;

[0032] The method further comprises:

[0033] When a synchronization success response message sent by the target cluster is received, the business operation record in the second queue is removed, and the business operation record stored on the disk is cleared.

[0034] Optionally, the concurrently transmitting the business operation records in the second queue to the target cluster based on the multi-channel network constructed between the source cluster and the target cluster includes at least one of the following:

[0035] Polling and querying each connection channel in the multi-channel network, and calling each connection channel in the multi-channel network in turn to send the business operation records in the second queue to the target cluster;

[0036] In the event that there is task blocking in the multi-channel network, the business operation records in the second queue are sent to the target cluster through a target connection channel in the multi-channel network; wherein the target connection channel is a connection channel with the least number of task blocking in the multi-channel network.

[0037] In a second aspect, an embodiment of the present application further provides a data synchronization device, applied to a source cluster, comprising:

[0038] A record reading module is used to read at least two business operation records pre-solidified on the disk to build a first queue;

[0039] an inspection and movement module, configured to inspect the business operation records in the first queue and move target business operation records that meet preset requirements to the second queue, wherein the target business operation record is at least one business operation record of the at least two business operation records;

[0040] A data synchronization module is used to send the business operation records in the second queue to the target cluster.

[0041] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a program stored on the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the data synchronization method as described in any one of the first aspects are implemented.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization method as described in any one of the first aspects are implemented.

[0043] In a fifth aspect, an embodiment of the present application further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the data synchronization method as described in any one of the first aspects.

[0044] In an embodiment of the present application, the business operation records pre-solidified on the disk are constructed as the first queue to ensure the basic guarantee of data persistence storage and integrity. Secondly, through the preset condition screening mechanism, the target operation records that meet the requirements are migrated to the second queue to achieve data filtering and priority stratification, effectively reducing the amount of invalid data transmission and improving network resource utilization. This embodiment implements step-by-step control of data synchronization through a dual-queue architecture, which not only ensures data reliability but also improves processing efficiency, while reducing the load pressure of the target cluster. The combination of solidified storage and dynamic screening takes into account the data persistence requirements and the flexibility of real-time synchronization. It is suitable for high-concurrency, low-latency data synchronization scenarios, has good scalability and system stability, can significantly reduce resource consumption and transmission delays in the data synchronization process, and improve overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flow chart of a data synchronization method in an embodiment of the present application;

[0047] Figure 2 This is a schematic diagram of a cross-cluster data synchronization method in related technologies;

[0048] Figure 3 is a schematic diagram of a data synchronization method in an embodiment of the present application;

[0049] Figure 4 Schematic diagram of the difference between order-preserving and non-order-preserving execution of cross-cluster synchronization operations in an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of a hardening process for a magnetic disk in an embodiment of the present application;

[0051] Figure 6 This is a schematic diagram of a source cluster reading business operation records from a disk in an embodiment of the present application;

[0052] Figure 7 This is a schematic diagram of a process for determining a second queue based on a first queue in an embodiment of the present application;

[0053] Figure 8 Schematic diagram of synchronous recording and sequence-preserving high-concurrency operations in an embodiment of the present application;

[0054] Figure 9 This is a schematic diagram of concurrent cross-cluster synchronization of multiple directories in an embodiment of the present application;

[0055] Figure 10 This is a schematic diagram of concurrent cross-cluster synchronization of multiple operation types in a single directory in an embodiment of the present application;

[0056] Figure 11 is a schematic diagram of a data synchronization device in an embodiment of the present application;

[0057] Figure 12 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] Please refer to Figure 1 , Figure 1 : is a schematic diagram of a data synchronization method in an embodiment of the present application. The data synchronization method is applied to a source cluster and specifically includes:

[0060] Step 101: Read at least two business operation records pre-solidified on a disk to construct a first queue.

[0061] It should be noted that multiple logical nodes (such as servers or virtual machines) are set in the source cluster, and each node can be bound to a specific CPU core, such as core0, core1. Each logical node reads the operation records on the disk through multiple channels (such as chn1 to chn8). In an embodiment of the present application, the basis for channel division can be the operation type (such as creation, setting permissions, etc.) and the CPU core number. For example, a business operation record "create operation" can be bound to core0, and the record of disk interval 1 can be read through chn1; a business operation record "set permission operation" can be bound to core1, and the record of disk interval 2 can be read through chn2.

[0062] In this embodiment, when writing operation records to disk, the disk range can be calculated based on the target name, operation type, and CPU core number, ensuring that records of the same type for the same target are stored sequentially. The first queue can then store read records in order to avoid duplicate reads.

[0063] In the above steps, the source cluster can read pre-hardened (i.e., persistently stored) business operation records from disk, such as create and delete operations. This can be done through multiple channels in parallel, with each channel bound to a specific CPU core, and tasks assigned based on operation type and CPU core number. This improves efficiency, reduces the overhead of CPU-bound thread switching, and enhances the reliability of hardened record data.

[0064] Step 102: Check the business operation records in the first queue and move target business operation records that meet preset requirements to the second queue, where the target business operation record is at least one of the at least two business operation records.

[0065] In step 102, for each record in the first queue, it can be determined whether each business operation record meets the preset requirements by comparing its operation type weight check, important task queue composition check, hash conflict check, etc. For example, if the current record is setting permissions (6 points), check whether there is a higher weight operation in front of it (such as creation 10 points). If so, it will not be processed for the time being; if not, it will be moved to the second queue. Alternatively, if the operation involves creation, renaming, etc., which may affect other operations, it is necessary to check the important task queue of its parent directory. Alternatively, a hash conflict detection is performed to determine whether the current operation conflicts with the operation of the same name in the queue by the hash value of the target name. For example, if there is creation A in the first queue, and the current record is deletion A, it is necessary to wait for creation A to be completed. Thus, records that meet the conditions (such as no high priority operation blocking, no same name conflict) are moved to the second queue.

[0066] It can be understood that the above-mentioned first queue is an intermediate queue for storing business operation records to be checked. Its core function is to verify the executability of operation records to ensure that only records that meet preset conditions can enter the subsequent synchronization process. It can be a check queue.

[0067] This way, the Check queue ensures that operations are executed in the correct order, avoiding data inconsistencies caused by incorrect priorities or dependencies. Furthermore, only records that meet the criteria are sent to the target cluster, which can be understood as the destination cluster, reducing invalid transmissions and improving synchronization efficiency. Furthermore, the Check queue supports dynamic priority adjustment (for example, prioritizing high-priority operations), adapting to complex business scenarios.

[0068] Step 103: Send the business operation records in the second queue to the target cluster.

[0069] It's important to note that the source cluster can concurrently send records in the second queue through multiple channels (for example, channel 1 to channel 128). Specifically, a round-robin scheduling (RR algorithm) can be used to send messages through each channel in sequence, such as channel 1 → channel 2 → ... → channel 128. This improves throughput through multi-channel concurrency, optimizes network resources through dynamic scheduling, and reduces latency in intermediate links through direct synchronization.

[0070] In the embodiment of the present application, the present application realizes efficient and reliable data synchronization through phased queue management and dynamic resource scheduling. Specifically, multi-logical nodes and multi-channels are used to read disk operation records in parallel, combined with CPU binding and disk interval calculation, to improve reading efficiency and reduce thread switching overhead, thereby ensuring data reliability. Furthermore, through the weight check, dependency verification and hash conflict detection of the Check queue, qualified records are dynamically screened to ensure that high-priority operations are executed first, avoid data conflicts and sequence errors, and improve synchronization reliability. Finally, multi-channel concurrent sending and polling scheduling algorithms are used to optimize network resource utilization, reduce latency, and improve throughput. The overall solution takes into account both performance and consistency through hierarchical queue management, dynamic priority control and resource isolation, adapts to the cross-cluster metadata synchronization needs in large-scale distributed scenarios, and significantly improves system stability and synchronization efficiency.

[0071] It is worth mentioning that Figure 2 and Figure 3As shown, compared with the cross-cluster data synchronization method in the related art, the embodiment of the present application adopts an asynchronous, message-order-preserving, and high-concurrency cross-cluster metadata synchronization method. The client sends the business instruction to the source cluster through Near Field Communication (NFC) / Common Internet File System (CIFS), and immediately returns a response to the front end after the source cluster processes the business request. The source cluster processes the business request asynchronously and synchronizes the metadata of the business request to the destination cluster according to the scanning algorithm, the order-preserving algorithm, and the high-concurrency cross-cluster. Among them, the source cluster contains a source cluster-logical node, and its core functions are: reading and queue construction of business operation records, that is, it can read at least two pre-solidified business operation records from the disk to build a first queue; screening and transfer of business operation records, that is, checking the operation records in the first queue, screening out records that meet the preset conditions, and moving the records that meet the conditions to the second queue for subsequent processing. It can be seen that the embodiment of the present application has a low impact on the front-end business performance, and the difference time between the two sets of cluster data is short, thereby achieving reliable and high-concurrency synchronization across clusters. In addition, the embodiments of the present application can be applied to the context of East Data West Storage in a wide area network environment, to achieve reliable and high-concurrency synchronization across clusters for East Data West Computing, improve the service performance of the source cluster, and ensure the accuracy and efficiency of cross-cluster synchronization services.

[0072] It should be noted that cross-cluster synchronization operations between two clusters must be executed in order, otherwise the results of the source and destination clusters may be different after executing the business operations. Figure 4 As shown, after the source cluster executes the business operation, there is no file in directory A; after the destination cluster executes the business operation, the file fileA exists in directory A. Therefore, the failure to maintain order causes the destination cluster to execute the business operation results that do not meet the expected results. In the embodiments of the present application, it is necessary to ensure that the operation dependencies are correctly satisfied to avoid data errors or logical anomalies caused by incorrect order.

[0073] Optionally, before reading the at least two business operation records pre-solidified on the disk, the method further includes:

[0074] Determine the CPU core number on which each of the business operation records is running based on the name, operation record type, and number of CPU cores of each of the business operation records;

[0075] Binding each of the business operation records to a corresponding CPU core number;

[0076] Determining the disk range corresponding to each business operation record according to the CPU core number corresponding to each business operation record;

[0077] According to the serial number of each business operation record, the serial number corresponding to each business operation record is written into the corresponding disk area in sequence.

[0078] It should be noted that the business operation records stored on disk are processed by the source cluster, which receives client metadata requests at the logical node granularity and records the operations to disk upon completing the business request. After processing the business message, the source cluster immediately responds to the requester without waiting for the destination cluster to complete the processing.

[0079] It is worth mentioning that Figure 5 As shown, the client sends service instructions to the source cluster via NFC / CIFS. NFC / CIFS acts as an intermediary layer, responsible for receiving and forwarding services. After receiving the service, the source cluster enters its internal processing flow. The source cluster's internal process consists of three key steps, executed in sequence: service operation processing, which performs preliminary processing on received service operations (such as parsing, validation, and parameter extraction); persistent storage of processed service operation records to disk to ensure they are not lost even if the system restarts abnormally; and transaction commit, which submits the persistent storage of service operation records to the transaction management system for final confirmation of the transaction. After completing the transaction commit, the source cluster can send a response to the client via NFC / CIFS, confirming the completion status (success or failure) of the service operation.

[0080] In a specific embodiment, the client's business operation records can be classified by the following solidification algorithm, and the corresponding CPU core number can be determined for each business operation record. Among them, the CPU core to which the current operation record should be bound can be calculated based on the name (name) of each business operation record, the type of operation record and the number of CPU cores of the central processing unit to execute the solidified record. In this way, different types of business operation records can be bound to the specified core of the CPU to work, so that the same operation records with the same target are on the same CPU core, and business operation records of different operation types with the same target are allocated to different CPU cores to work, so as to achieve equal distribution of resources among the tasks on each CPU core and reduce the blocking effect of a large number of low-priority operations on high-priority records.

[0081] For example, based on the target name, operation type, and the number of system CPU cores, the CPU core ID of each operation record is calculated. That is, it can be expressed as:

[0082] "while(c=*target name++)

[0083] {

[0084] key=(key<<4)|(key>>(8*sizeof(key)-4));

[0085] key = key^c;

[0086] }

[0087] type = (create|delete)? 1: (rename)? 2: (link)? 3: (set permissions)? 4: (set attributes)? 5: (allocate segment)? 6: 7;

[0088] key+=operation type type;

[0089] key=key^(key>>3)^(key>>5);

[0090] coreid (cpu core number) = key% cores (cpu core number)"

[0091] It is understood that before each operation record is solidified to disk, a unique serial number (sn) is generated based on its CPU core number and operation type. For example, the create operation of core0 generates sn=1001, and the set permission operation of core1 generates sn=1002. Therefore, the storage location on the disk (e.g., disk interval 1, disk interval 2) can be calculated based on the name, operation type, and CPU core number of the operation record. For example, the create operation is bound to core0 and written to disk interval 1; the set permission operation is bound to core1 and written to disk interval 2. In this way, the same type of operation records for the same target are written to the disk in serial number order, ensuring that no additional sorting is required when reading.

[0092] In this way, the source cluster uniformly processes the business operation records to be synchronized through the above solidification algorithm, and obtains the business operation records solidified on the disk. After the source cluster starts the background thread task, it can synchronize the business operations across clusters to the destination cluster in an asynchronous manner, facilitating the order-preserving and high-concurrency processing of asynchronously synchronized operation records.

[0093] Optionally, reading the at least two business operation records recorded on the disk to construct a first queue includes:

[0094] Concurrently scanning a specified interval in the disk through multiple channels to obtain channel queues corresponding to multiple scanning threads; wherein the multiple scanning threads are threads formed by the source cluster scanning the specified interval through the multiple channels, and the channel queue includes at least one business operation record located in the specified interval;

[0095] Obtaining an operation type or a corresponding CPU core number of a first business operation record in each channel queue, where the first business operation record is the business operation record arranged at the head of the channel queue, and the sequence number of the first business operation record is the smallest sequence number among the sequence numbers of multiple business operation records in the channel queue where the first business operation record is located;

[0096] Based on the operation type of the first business operation record in the channel queues corresponding to each of the multiple scanning threads or the CPU core number corresponding to the first business operation record, a first operation is performed to obtain the first queue.

[0097] In some embodiments, when reading business operation records on a disk and constructing a first queue, efficient and orderly data processing can be achieved through multi-channel concurrent scanning and dynamic merging strategies.

[0098] Specifically, if Figure 6 As shown, the source cluster can use multiple channels for its logical nodes to concurrently scan specified disk intervals. Each channel corresponds to a scanning thread, responsible for reading operation records from a specific interval on the disk. For example, chn1 scans disk interval 1, chn2 scans disk interval 2, and so on. Each scanning thread then stores the read records in sequence number order into the corresponding channel queue (e.g., chn1_queue, chn2_queue). The channel queue contains multiple business operation records, and the first record in the queue has the smallest sequence number (i.e., the record that was first persisted to disk).

[0099] Then, for each channel queue, you can extract the business operation record at the head of the queue (i.e., the record with the smallest sequence number). For example, the head record of chn1_queue is rec1 (operation type: create, CPU core number: core0). The head record of chn2_queue is rec2 (operation type: set permissions, CPU core number: core1). Specifically, you can obtain the operation type (e.g., create, set permissions) or the corresponding CPU core number (e.g., core0, core1) of the head record.

[0100] The source cluster can merge or filter the head records of multiple channel queues based on the operation type or CPU core number of the business operation record. Exemplarily, it can be merged by operation type. If the head record of chn1_queue is creation (operation type) and the head record of chn2_queue is setting permissions, the creation record can be added to the first queue first. Exemplarily, it can be merged by CPU core number. If the head record of chn1_queue is bound to core0 and the head record of chn2_queue is bound to core1, they are merged to the first queue in order of CPU core number. If the operation type is high priority (such as creation), it will be processed first; if the operation type is low priority (such as setting permissions), wait for the higher priority operation to complete.

[0101] Therefore, the embodiment of the present application can add records that meet the conditions (such as high-priority operations) to the first queue (such as the check queue) in sequence for subsequent inspection and synchronization.

[0102] In this way, the embodiments of the present application enable efficient reading of business operation records through multi-channel concurrent scanning and dynamic merging strategies, enabling multi-channel parallel scanning of disk intervals to improve data acquisition speed. Dynamic priority control is also implemented, merging records based on operation type or CPU core number to optimize resource utilization and avoid blocking low-priority operations. This lays a foundation for efficient and orderly subsequent check queue inspections and synchronization operations, significantly improving the performance and stability of cross-cluster metadata synchronization.

[0103] Optionally, the first operation includes at least one of the following:

[0104] When the operation type of the first business operation record in each channel queue is consistent, the business operation record in each channel queue is sequentially added to a preset queue to obtain the first queue;

[0105] In the case that there is at least one channel queue in which the operation type of the first business operation record is the target operation type, the first business operation record with the operation type of the target operation type is added to the preset queue to obtain the first queue.

[0106] In some embodiments, after multiple channels concurrently scan a specified interval of the disk and generate multiple channel queues (e.g., chn1_queue, chn2_queue), the system dynamically adjusts the construction logic of the preset queue based on the consistency of the operation type or the target operation type. In a specific embodiment, the head record operation type of all channel queues is consistent (e.g., all are setting attributes, allocating data segments, etc.). For example, the head record of chn1_queue is create (operation type create), and the head record of chn2_queue is also create, and the head record operation type of all channel queues is create.

[0107] In a specific embodiment, all business operation records in each channel queue can be added to a preset queue (such as a check queue) in sequence according to the order of the channel queues (such as chn1_queue→chn2_queue→...). For example, chn1_queue contains rec1 (setting attributes) and rec2 (setting attributes), and chn2_queue contains rec3 (setting attributes) and rec4 (setting attributes). After the merge, the check queue is added to rec1, rec2, rec3, and rec4 in sequence. Since the disk is sorted by operation type and CPU core number during solidification, the records in the channel queue are themselves ordered (in ascending order by serial number). After the merge, the records in the preset queue maintain consistent operation types and correct order, without the need for additional sorting. In this way, the embodiment of the present application can effectively reduce the sorting overhead. If the operation types are consistent, there is no need for dynamic sorting, and they can be directly merged in the order of channel queues, saving computing resources and improving processing efficiency, so that high-priority operations can quickly enter the subsequent inspection process, avoiding blocking of low-priority operations, and also achieving data consistency protection. In the merged preset queues, the records of the same operation type are in the correct order, ensuring the reliability of the synchronization logic.

[0108] In another specific embodiment, the target operation type can be processed with priority, and the operation type of the first record of at least one channel queue is the target operation type (such as high-priority operations such as create and delete). For example, the first record of chn1_queue is create (operation type create), the first record of chn2_queue is set permission (operation type set_perm), and there is at least one channel queue whose first record operation type is create. Filter the target operation type records, traverse all channel queues, and extract the first record of the target operation type. For example, the first record of chn1_queue is create (create), and the first record of chn2_queue is set permission (set_perm), and only add the create operation record to the preset queue. If there are multiple channel queues whose first records are the target operation type, sort them by priority (such as create > delete > set permission) and select the record with the highest priority. For example, the first record of chn1_queue is create (priority 10), and the first record of chn2_queue is delete (priority 7), and the create operation record is selected first.

[0109] In this way, by filtering the target operation type, high-priority operations (such as creation) can be ensured to quickly enter the subsequent process, avoiding blocking of low-priority operations. The queue content can also be dynamically adjusted according to the operation type, optimizing resource utilization and improving synchronization efficiency. High-priority operations (such as creation) can also be executed first to ensure that the dependencies of subsequent operations (such as deletion) are met.

[0110] Optionally, checking the business operation records in the first queue and moving target business operation records that meet preset requirements to the second queue includes:

[0111] assigning a corresponding weight to each business operation record in the first queue according to an operation type of each business operation record in the first queue;

[0112] If a second business operation record is arranged after a third business operation record in the first queue and a weight of the second business operation record is less than or equal to a weight of the third business operation record, determining that the state of the second business operation record is a first state; wherein the second business operation record and the third business operation record are any two business operation records in the first queue, and processing of the second business operation record is suspended in the first state;

[0113] If the fourth business operation record is arranged after the third business operation record in the first queue and the weight of the fourth business operation record is greater than the weight of the third business operation record, determining that the status of the fourth business operation record is the second status;

[0114] If the fourth business operation record is in the second state, determining, based on an operation type of the fourth business operation record and a pre-acquired hash value, whether the fourth business operation record has been classified into a third queue, the third queue being used to store business operation records related to a second operation, the second operation including at least one of creation, deletion, rename, hard link, and soft link;

[0115] If the fourth business operation record is not included in the third queue, the fourth business operation record is moved to the second queue.

[0116] In some embodiments, such as Figure 7 and Figure 8 As shown, dynamic weight allocation, status judgment and hash value verification can be used to achieve intelligent screening and processing of business operation records in the first queue (Check queue), ensuring that high-priority operations are executed first while avoiding blocking of low-priority operations. Specifically, according to the operation type of the business operation record (such as create, delete, rename, etc.), it can be assigned a preset weight (for example, 10 points for create, 7 points for delete, 6 points for setting permissions). For example, the weight of the create operation record is 10, the weight of the delete operation record is 7, and the weight of the set permission operation record is 6. In this way, the urgency of the operation can be distinguished by weight to ensure that high-priority operations (such as create) are processed first. In addition, the weight can be flexibly configured to meet the needs of different business scenarios.

[0117] In a specific embodiment, Figure 7 As shown, disk records are divided into multiple independent channels (for example, chn1 to chn8), and each channel contains a set of records (rec1, rec2, ..., recn). Each channel reads 128 records at a time, and the reading range is marked with a blue dotted box to reduce the number of I / O operations and improve efficiency. The record at the head of the queue (that is, the earliest record to be processed) can be taken from the queue of each channel. In addition, the serial numbers (sn) of the head records of all channel queues are compared, and the record with the smallest sn is selected as the next processing object. Subsequently, the selected record can be removed from the queue of the channel to which it belongs. For example, after the record of channel chn8 is processed, it is deleted from the queue of that channel. In this way, the processed records can be added to the "check queue" for use in subsequent steps.

[0118] In another specific embodiment, Figure 8 As shown in the figure, the source cluster is responsible for reading business operation records from the local disk and classifying, sorting, and synchronizing them according to the rules. The specific steps are as follows:

[0119] Step 201: Each logical node in the source cluster reads the business operation record in the disk;

[0120] In this step, the logical nodes of the source cluster read all business operation records from the disk. Business operation records stored on the disk may contain various types (such as file operations and configuration changes) and need to be processed according to the rules.

[0121] Step 202: Each business operation record is sorted according to its sequence number (sn) and added to the first queue;

[0122] Specifically, the read business operation records can be sorted by sequence number (sn) to ensure the correctness of the processing order. The sequence number (sn) is used to identify the order of business operations to avoid disorder.

[0123] Step 203: Synchronous records of the same parent directory are added to the third queue of the parent directory;

[0124] It is understood that business operation records with the same parent directory can be classified into the corresponding third queue. Thus, grouping by parent directory may be used to optimize synchronization efficiency (for example, files in the same directory are synchronized more efficiently). The third queue can be a sub-queue divided by directory for subsequent refined processing.

[0125] Step 204: Whether the business operation records in the first queue are allowed to be synchronized concurrently;

[0126] This step can be used to check whether concurrent synchronization is allowed for the business operation records in the first queue. Whether to enable concurrent processing is determined based on business rules or system load, such as record type, resource usage, and system configuration.

[0127] Step 205: If the business operation records in the first queue are allowed to be synchronized concurrently, synchronize the business operation records in the first queue and add them to the second queue;

[0128] In this step, if concurrent synchronization is allowed, the records in the first queue can be transferred to the second queue. The second queue can be a concurrent processing queue for executing synchronization tasks in parallel. This facilitates improving overall synchronization efficiency through concurrent processing.

[0129] Step 206: The business operation records in the second queue are concurrently synchronized to the remote end;

[0130] In this step, the business operation records in the second queue are synchronized to the target cluster. Specifically, data can be transmitted through a network protocol (such as TCP / IP, HTTP, etc.). In this embodiment, the synchronization speed can be improved through multi-threading or distributed task scheduling.

[0131] Step 207: The target cluster processes the business operation records in the second queue.

[0132] Specifically, the target cluster receives and processes synchronized data from the source cluster. The target cluster's logical nodes process the business operation records synchronized from the source cluster. This processing may include file writes, configuration updates, and status updates. If processing fails, retries or logging may be required.

[0133] For the first state, in the first queue, if the second business operation record (rec2) is arranged after the third business operation record (rec3), and the weight of rec2 is less than or equal to the weight of rec3, then the state of rec2 is the first state, that is, the state of suspended processing. For example, the weight of rec2 is 6 (setting permissions), the weight of rec3 is 7 (deletion), the weight of rec2 ≤ the weight of rec3 → rec2 suspends processing. For the second state, in the first queue, if the fourth business operation record (rec4) is arranged after the third business operation record (rec3), and the weight of rec4 is greater than the weight of rec3, then the state of rec4 is the second state, that is, the state that requires further verification. For example, the weight of rec4 is 10 (creation), the weight of rec3 is 7 (deletion), the weight of rec4 > the weight of rec3 → rec4 enters the second state. In this way, the embodiment of the present application avoids low-priority operations blocking high-priority operations through weight comparison, ensures the correctness of the synchronization order, and can divide operation records into two categories: "pause processing" and "need to be verified", thereby improving processing efficiency.

[0134] Furthermore, for rec4 in the second state, it can be determined whether it has been included in the third queue (used to store records related to the second operation) based on its operation type (e.g., create, delete, etc.) and a pre-acquired hash value (e.g., the hash value of the target name). For example, rec4's operation type is create, the target name is A, and the hash value is hash(A). If hash(A) already exists in the third queue, rec4 may conflict with other operations in the queue (e.g., delete A is not completed).

[0135] The role of the third queue is to store records related to the second operation (such as create, delete, rename, etc.) to ensure that the dependencies of these operations are correctly processed. For example, if rec4 is a create A operation and there is already a delete A operation in the third queue, it is necessary to wait for delete A to complete before processing create A. In this way, the embodiment of the present application can avoid conflicting operations (such as create and delete) of the same target from being executed simultaneously through hash value verification. The third queue ensures that high-priority operations are executed first to avoid blocking low-priority operations.

[0136] Therefore, if rec4 is not included in the third queue (i.e., it does not conflict with other operations in the queue), it is moved to the second queue (Send queue) and awaits subsequent synchronization to the target cluster. For example, rec4 is a Create A operation, and there is no Delete A operation in the third queue → rec4 is moved to the Send queue.

[0137] It's worth noting that the second queue can be a temporary storage queue for business operation records that have been checked and meet the conditions, for subsequent transmission to the target cluster. The second queue is a collection of business operation records that meet the preset conditions after being checked by the first queue (the Check queue). These records have passed processes such as weight checking, dependency verification, and hash collision detection to ensure that they can be synchronized to the target cluster safely and orderly. The records in the second queue are verified and executable operation records, ensuring the reliability of the synchronization process.

[0138] In this way, the above embodiment can dynamically assign priorities based on operation type, ensuring that high-priority operations are executed first. It can also classify operation records into two categories, "paused processing" and "requires verification," through weight comparison, to avoid blocking low-priority operations. Furthermore, the operation type and hash value can be combined to determine whether a record conflicts with other operations in the queue, ensuring data consistency. As a result, only records that meet the criteria can be moved to the second queue, reducing invalid transmissions and improving synchronization efficiency.

[0139] Optionally, the method further includes:

[0140] If the fourth business operation record has been included in the third queue, determining whether a fifth business operation record exists in the third queue based on a hash value of the fourth business operation record; wherein the name of the fifth business operation record is the same as the name of the fourth business operation record;

[0141] If the fifth business operation record exists in the third queue and synchronization processing of the fifth business operation record has not been completed, move the fourth business operation record to the second queue;

[0142] When the fifth business operation record does not exist in the third queue, the fourth business operation record is moved to the second queue.

[0143] In some embodiments, this embodiment implements dynamic processing of business operation records through hash value verification and queue status judgment, ensuring that high-priority operations are executed first while avoiding conflicting operations on the same target.

[0144] It is worth mentioning that the third queue is used to store business operation records related to the second operation (such as creation, deletion, renaming, hard link, soft link). These operations usually involve changes or dependencies on the target name, and their execution order needs to be strictly managed. For the fourth business operation record (rec4), the system determines whether it has been included in the third queue based on its operation type and the hash value of the target name. For example, rec4 is an operation to create A, the target name is A, and the hash value is hash(A). If hash(A) already exists in the third queue, it means that there are other operations related to A (such as deleting A or renaming A).

[0145] When rec4 has been included in the third queue, the system checks whether there is a fifth business operation record with the same name (recorded as rec5) in the third queue. For example, rec4 is an operation to create A, and there is rec5 (an operation to delete A) in the third queue, and the name of rec5 is the same as rec4 (both are A). If rec5 exists and the synchronization processing is not completed, rec4 needs to wait for rec5 to be completed before execution, but according to the preset logic, rec4 will still be moved to the second queue (Send queue). For example, rec4 is to create A, rec5 is to delete A, and rec5 has not completed synchronization → rec4 is moved to the Send queue. If rec5 does not exist, rec4 is directly moved to the Send queue without waiting.

[0146] Subsequently, regardless of whether rec5 exists in the third queue, as long as rec4 has been included in the third queue and meets one of the following conditions: rec5 exists but the synchronization process is not completed; rec5 does not exist. rec4 will be moved to the second queue (Send queue) and wait for subsequent synchronization to the target cluster. For example, rec4 creates A, and there is rec5 in the third queue (deletes A, synchronization is not completed) → rec4 is moved to the Send queue. rec4 creates A, and there is no rec5 in the third queue → rec4 is moved to the Send queue. In this way, in the embodiment of the present application, even if there is a potential conflict, the source cluster still allows rec4 to enter the Send queue, reducing blocking time and improving synchronization efficiency. The correct order of operations is ensured through subsequent synchronization logic (such as weight checking and dependency verification) to avoid data inconsistency.

[0147] In this way, the embodiment of the present application can determine whether there is an operation record with the same name through the hash value of the target name to avoid conflicts, and when there is an operation record with the same name in the third queue, the current operation record is still allowed to enter the Send queue, but relies on subsequent synchronization logic to ensure the correct order. It can effectively handle complex operation dependencies and avoid data conflicts, while optimizing resource utilization and improving system stability.

[0148] Optionally, sending the business operation record in the second queue to the target cluster includes:

[0149] Based on the multi-channel network constructed between the source cluster and the target cluster, concurrently sending the business operation records in the second queue to the target cluster;

[0150] The method further comprises:

[0151] When a synchronization success response message sent by the target cluster is received, the business operation record in the second queue is removed, and the business operation record stored on the disk is cleared.

[0152] In some embodiments, efficient and reliable data synchronization can be achieved through multi-channel network concurrent sending and cleanup logic after successful synchronization. Specifically, multiple independent channels (such as channel1 to channel128) are established between the source cluster and the target cluster, and each channel corresponds to an independent network connection. Channels are dynamically allocated based on the operation type (such as create, delete) of the business operation record or the hash value of the target name. For example, the create operation is assigned to channel1 and the delete operation is assigned to channel2, ensuring that different operation types use different channels to avoid resource competition. Subsequently, the business operation records can be sent using each channel in sequence, for example, channel1→channel2→...→channel128.

[0153] Furthermore, if a channel is busy (e.g., channel 1 has 5 blocked tasks), the system selects the channel with the least blocked tasks (e.g., channel 2) to assign the new task. This effectively improves throughput, and by sending data across multiple channels in parallel, it significantly reduces synchronization time, avoids overloading a single channel, and improves network utilization.

[0154] In some other embodiments, after receiving the business operation record, the target cluster sends a synchronization success response message to the source cluster. The source cluster needs to confirm the integrity of the response information (such as check code, serial number) to ensure that the data is received correctly. Remove the records in the second queue. After receiving the response information, the source cluster removes the synchronized business operation records from the second queue (Send queue). In addition, the records stored on the disk can be cleared, that is, the source cluster deletes the synchronized business operation records from the disk to avoid repeated processing. For example, if rec1 (create A) has been successfully synchronized to the target cluster, the source cluster removes rec1 from the Send queue and deletes the solidified record of rec1 from the disk. In this way, the embodiment of the present application can clean up the records immediately after the synchronization is successful, ensure that the data status of the source cluster is consistent with that of the target cluster, reduce disk storage pressure, avoid the accumulation of redundant data, and ensure that the data is correctly transmitted through the response mechanism to avoid data loss due to network failure.

[0155] Optionally, the concurrently transmitting the business operation records in the second queue to the target cluster based on the multi-channel network constructed between the source cluster and the target cluster includes at least one of the following:

[0156] Polling and querying each connection channel in the multi-channel network, and calling each connection channel in the multi-channel network in turn to send the business operation records in the second queue to the target cluster;

[0157] In the event that there is task blocking in the multi-channel network, the business operation records in the second queue are sent to the target cluster through a target connection channel in the multi-channel network; wherein the target connection channel is a connection channel with the least number of task blocking in the multi-channel network.

[0158] In some embodiments, efficient concurrent sending of business operation records in a multi-channel network can be achieved through polling scheduling and dynamic task allocation. The source cluster can traverse each connection channel in the multi-channel network in order (for example, channel1→channel2→...→channelN). Each channel is called in turn to send the business operation records in the second queue to the target cluster in order. For example, there are rec1, rec2, and rec3 in the second queue, and the channels are channel1, channel2, and channel3. rec1 is sent through channel1, rec2 is sent through channel2, and rec3 is sent through channel3. If a channel task is busy (for example, channel1 has 5 tasks blocked), the system selects the channel with the least task blocking (such as channel2) to invest in the new task. In addition, after the target cluster confirms that the synchronization is successful, the source cluster removes the record from the send queue and clears the operation record on the disk.

[0159] Furthermore, the source cluster can monitor the task congestion status (e.g., task queue length, response latency) of each connected channel in the multi-channel network in real time. When task congestion is detected on certain channels, the channel with the fewest blocked tasks (i.e., the channel with the lowest load) is selected to send the business operation record. For example, if channel 1 has 5 blocked tasks, channel 2 has 2 blocked tasks, and channel 3 has 0 blocked tasks, then channel 3 is selected to send the new task.

[0160] It can be seen that the polling scheduling in the embodiment of the present application is suitable for normal load scenarios to ensure fairness, and the dynamic task allocation is suitable for scenarios with high load or large differences in channel performance to improve data synchronization efficiency.

[0161] In one embodiment, Figure 9 and Figure 10 As shown in the figure, it shows the concurrent synchronization mechanism of cross-cluster business operation records, focusing on how business operations between different directories and different targets in the same directory can be efficiently synchronized through a multi-channel network.

[0162] exist Figure 9 During concurrent cross-directory synchronization, the source cluster contains three directories (A, B, and C), each containing a business operation record (such as "creating filedA"). For example: Directory A (creating filedA), Directory B (creating filedA), and Directory C (creating filedA). The target cluster contains three corresponding directories (A, B, and C), each receiving synchronization operation records from the source cluster. The source cluster concurrently sends these operation records from the three directories to the target cluster via a multi-channel network. Each directory's operation record is processed independently, without interfering with each other.

[0163] Operations on directories A, B, and C in the source cluster can be sent simultaneously to the corresponding directories in the target cluster, without waiting for operations on other directories to complete. For example, the creation of directory A (filedA) and directory B (filedA) can be processed in parallel, improving overall synchronization efficiency. The directory structure of the source and target clusters remains consistent (A→A, B→B, C→C), ensuring synchronization accuracy.

[0164] exist Figure 10 In the example, the source cluster (directory A) contains three business operation records (business operation records 1, 2, and 3), all of which are for creating filedA. The target cluster (directory A) receives the three business operation records from the source cluster and stores them separately. The source cluster concurrently sends the three operation records to directory A of the target cluster through a multi-channel network, achieving concurrency for different targets under the same directory. Multiple business operation records in directory A of the source cluster (such as creating filedA) can be sent to directory A of the target cluster at the same time without having to be processed sequentially. For example, business operation records 1, 2, and 3 can be synchronized in parallel to reduce waiting time. As a result, each operation record is processed independently without interfering with each other, ensuring stability in high-concurrency scenarios.

[0165] Please refer to Figure 11 In an embodiment of the present application, a data synchronization device 400 is applied to a source cluster and includes:

[0166] The record reading module 401 is used to read at least two business operation records pre-solidified on the disk and build a first queue;

[0167] An inspection and movement module 402 is configured to inspect the business operation records in the first queue and move a target business operation record that meets preset requirements to a second queue, wherein the target business operation record is at least one of the at least two business operation records;

[0168] The data synchronization module 403 is configured to send the business operation records in the second queue to the target cluster.

[0169] Optionally, before reading the at least two business operation records pre-solidified on the disk, the data synchronization device 400 further includes:

[0170] A first determining module is configured to determine the CPU core number on which each of the business operation records is running based on the name, operation record type, and number of CPU cores of each of the business operation records;

[0171] A record binding module, configured to bind each of the business operation records to a corresponding CPU core number;

[0172] The interval corresponding module is used to determine the disk interval corresponding to each business operation record according to the CPU core number corresponding to each business operation record;

[0173] The record writing module is used to write the serial number corresponding to each business operation record into the corresponding disk area in sequence according to the serial number of each business operation record.

[0174] Optionally, the record reading module 401 is used to:

[0175] Concurrently scanning a specified interval in the disk through multiple channels to obtain channel queues corresponding to multiple scanning threads; wherein the multiple scanning threads are threads formed by the source cluster scanning the specified interval through the multiple channels, and the channel queue includes at least one business operation record located in the specified interval;

[0176] Obtaining an operation type or a corresponding CPU core number of a first business operation record in each channel queue, where the first business operation record is the business operation record arranged at the head of the channel queue, and the sequence number of the first business operation record is the smallest sequence number among the sequence numbers of multiple business operation records in the channel queue where the first business operation record is located;

[0177] Based on the operation type of the first business operation record in the channel queues corresponding to each of the multiple scanning threads or the CPU core number corresponding to the first business operation record, a first operation is performed to obtain the first queue.

[0178] Optionally, the first operation includes at least one of the following:

[0179] When the operation type of the first business operation record in each channel queue is consistent, the business operation record in each channel queue is sequentially added to a preset queue to obtain the first queue;

[0180] In the case that there is at least one channel queue in which the operation type of the first business operation record is the target operation type, the first business operation record with the operation type of the target operation type is added to the preset queue to obtain the first queue.

[0181] Optionally, the inspection movement module 402 is used to:

[0182] assigning a corresponding weight to each business operation record in the first queue according to an operation type of each business operation record in the first queue;

[0183] If a second business operation record is arranged after a third business operation record in the first queue and a weight of the second business operation record is less than or equal to a weight of the third business operation record, determining that the state of the second business operation record is a first state; wherein the second business operation record and the third business operation record are any two business operation records in the first queue, and processing of the second business operation record is suspended in the first state;

[0184] If the fourth business operation record is arranged after the third business operation record in the first queue and the weight of the fourth business operation record is greater than the weight of the third business operation record, determining that the status of the fourth business operation record is the second status;

[0185] If the fourth business operation record is in the second state, determining, based on an operation type of the fourth business operation record and a pre-acquired hash value, whether the fourth business operation record has been classified into a third queue, the third queue being used to store business operation records related to a second operation, the second operation including at least one of creation, deletion, rename, hard link, and soft link;

[0186] If the fourth business operation record is not included in the third queue, the fourth business operation record is moved to the second queue.

[0187] Optionally, the data synchronization device 400 further includes:

[0188] a second determining module, configured to determine, if the fourth business operation record has been classified into the third queue, whether a fifth business operation record exists in the third queue based on a hash value of the fourth business operation record; wherein the name of the fifth business operation record is the same as the name of the fourth business operation record;

[0189] a first moving module, configured to move the fourth business operation record to the second queue if the fifth business operation record exists in the third queue and synchronization processing of the fifth business operation record has not been completed;

[0190] The second moving module is configured to move the fourth business operation record to the second queue if the fifth business operation record does not exist in the third queue.

[0191] Optionally, the data synchronization module 403 includes:

[0192] a concurrency unit, configured to concurrently send the business operation records in the second queue to the target cluster based on a multi-channel network constructed between the source cluster and the target cluster;

[0193] The data synchronization device 400 is further configured to:

[0194] When a synchronization success response message sent by the target cluster is received, the business operation record in the second queue is removed, and the business operation record stored on the disk is cleared.

[0195] Optionally, the concurrency unit is used for at least one of the following:

[0196] Polling and querying each connection channel in the multi-channel network, and calling each connection channel in the multi-channel network in turn to send the business operation records in the second queue to the target cluster;

[0197] In the event that there is task blocking in the multi-channel network, the business operation records in the second queue are sent to the target cluster through a target connection channel in the multi-channel network; wherein the target connection channel is a connection channel with the least number of task blocking in the multi-channel network.

[0198] The data synchronization device 400 provided in the embodiment of the present application can perform the above Figure 1 The implementation principle and technical effects of the method embodiment shown are similar, and will not be described in detail in this embodiment.

[0199] The present application also provides an electronic device. Since the principle of solving the problem of the electronic device is similar to the image detection model training method in the present application, the implementation of the electronic device can refer to Figure 12 The implementation of the method shown in FIG. 1 is omitted for repetition. Figure 12 As shown, the electronic device 500 of the embodiment of the present application includes: a processor 510, which is used to read the program in the memory 520 and execute the following process:

[0200] Read at least two business operation records pre-solidified on the disk to build a first queue;

[0201] Checking the business operation records in the first queue and moving target business operation records that meet preset requirements to the second queue, the target business operation record being at least one business operation record of the at least two business operation records;

[0202] Sending the service operation records in the second queue to the target cluster through the transceiver 530;

[0203] The transceiver 530 is configured to receive and send data under the control of the processor 510 .

[0204] Among them, Figure 12In the embodiment of the present invention, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits such as one or more processors represented by processor 510 and memory represented by memory 520. The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are not further described herein. The bus interface provides an interface.

[0205] Optionally, the processor 510 is further configured to read a program in the memory 520 and perform the following steps:

[0206] Determine the CPU core number on which each of the business operation records is running based on the name, operation record type, and number of CPU cores of each of the business operation records;

[0207] Binding each of the business operation records to a corresponding CPU core number;

[0208] Determining the disk range corresponding to each business operation record according to the CPU core number corresponding to each business operation record;

[0209] According to the serial number of each business operation record, the serial number corresponding to each business operation record is written into the corresponding disk area in sequence.

[0210] Optionally, the processor 510 is further configured to read a program in the memory 520 and execute the following steps:

[0211] Concurrently scanning a specified interval in the disk through multiple channels to obtain channel queues corresponding to multiple scanning threads; wherein the multiple scanning threads are threads formed by the source cluster scanning the specified interval through the multiple channels, and the channel queue includes at least one business operation record located in the specified interval;

[0212] Obtaining an operation type or a corresponding CPU core number of a first business operation record in each channel queue, where the first business operation record is the business operation record arranged at the head of the channel queue, and the sequence number of the first business operation record is the smallest sequence number among the sequence numbers of multiple business operation records in the channel queue where the first business operation record is located;

[0213] Based on the operation type of the first business operation record in the channel queues corresponding to each of the multiple scanning threads or the CPU core number corresponding to the first business operation record, a first operation is performed to obtain the first queue.

[0214] Optionally, the first operation includes at least one of the following:

[0215] When the operation type of the first business operation record in each channel queue is consistent, the business operation record in each channel queue is sequentially added to a preset queue to obtain the first queue;

[0216] In the case that there is at least one channel queue in which the operation type of the first business operation record is the target operation type, the first business operation record with the operation type of the target operation type is added to the preset queue to obtain the first queue.

[0217] Optionally, the processor 510 is further configured to read a program in the memory 520 and execute the following steps:

[0218] assigning a corresponding weight to each business operation record in the first queue according to an operation type of each business operation record in the first queue;

[0219] If a second business operation record is arranged after a third business operation record in the first queue and a weight of the second business operation record is less than or equal to a weight of the third business operation record, determining that the state of the second business operation record is a first state; wherein the second business operation record and the third business operation record are any two business operation records in the first queue, and processing of the second business operation record is suspended in the first state;

[0220] If the fourth business operation record is arranged after the third business operation record in the first queue and the weight of the fourth business operation record is greater than the weight of the third business operation record, determining that the status of the fourth business operation record is the second status;

[0221] If the fourth business operation record is in the second state, determining, based on an operation type of the fourth business operation record and a pre-acquired hash value, whether the fourth business operation record has been classified into a third queue, the third queue being used to store business operation records related to a second operation, the second operation including at least one of creation, deletion, rename, hard link, and soft link;

[0222] If the fourth business operation record is not included in the third queue, the fourth business operation record is moved to the second queue.

[0223] Optionally, the processor 510 is further configured to read a program in the memory 520 and perform the following steps:

[0224] If the fourth business operation record has been included in the third queue, determining whether a fifth business operation record exists in the third queue based on a hash value of the fourth business operation record; wherein the name of the fifth business operation record is the same as the name of the fourth business operation record;

[0225] If the fifth business operation record exists in the third queue and synchronization processing of the fifth business operation record has not been completed, move the fourth business operation record to the second queue;

[0226] When the fifth business operation record does not exist in the third queue, the fourth business operation record is moved to the second queue.

[0227] Optionally, the processor 510 is further configured to read a program in the memory 520 and execute the following steps:

[0228] Based on the multi-channel network constructed between the source cluster and the target cluster, concurrently sending the service operation records in the second queue to the target cluster through the transceiver 530;

[0229] Also perform the following steps:

[0230] When a synchronization success response message sent by the target cluster is received, the business operation record in the second queue is removed, and the business operation record stored on the disk is cleared.

[0231] Optionally, the processor 510 is further configured to read a program in the memory 520 and execute at least one of the following:

[0232] Polling and querying each connection channel in the multi-channel network, and calling each connection channel in the multi-channel network in turn to send the business operation records in the second queue to the target cluster;

[0233] In the event that there is task blocking in the multi-channel network, the business operation records in the second queue are sent to the target cluster through a target connection channel in the multi-channel network; wherein the target connection channel is a connection channel with the least number of task blocking in the multi-channel network.

[0234] The electronic device 500 provided in the embodiment of the present application can perform the above Figure 1 The implementation principle and technical effects of the method embodiment shown are similar, and will not be described in detail in this embodiment.

[0235] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned Figure 1The various processes of the data synchronization method embodiment in the embodiment can achieve the same technical effect, and are not repeated here to avoid repetition. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0236] The embodiment of the present application further provides a computer program / program product, which is stored in a storage medium and is executed by at least one processor to implement the above Figure 1 The various processes of the data synchronization method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection of some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0238] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may be physically included separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0239] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute some steps of the sending and receiving methods described in various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0240] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data synchronization method, characterized in that: Applied to the source cluster, the method includes: Read at least two business operation records pre-solidified on the disk to build a first queue; Checking the business operation records in the first queue and moving target business operation records that meet preset requirements to the second queue, the target business operation record being at least one business operation record of the at least two business operation records; The business operation records in the second queue are sent to the target cluster.

2. The method according to claim 1, characterized in that Before reading the at least two business operation records pre-solidified on the disk, the method further includes: Determine the CPU core number on which each of the business operation records is running based on the name, operation record type, and number of CPU cores of each of the business operation records; Binding each of the business operation records to a corresponding CPU core number; Determining the disk range corresponding to each business operation record according to the CPU core number corresponding to each business operation record; According to the serial number of each business operation record, the serial number corresponding to each business operation record is written into the corresponding disk area in sequence.

3. The method according to claim 1 or 2, characterized in that The reading of the at least two business operation records recorded on the disk to construct a first queue includes: Concurrently scanning a specified interval in the disk through multiple channels to obtain channel queues corresponding to multiple scanning threads; wherein the multiple scanning threads are threads formed by the source cluster scanning the specified interval through the multiple channels, and the channel queue includes at least one business operation record located in the specified interval; Obtaining an operation type or a corresponding CPU core number of a first business operation record in each channel queue, where the first business operation record is the business operation record arranged at the head of the channel queue, and the sequence number of the first business operation record is the smallest sequence number among the sequence numbers of multiple business operation records in the channel queue where the first business operation record is located; Based on the operation type of the first business operation record in the channel queues corresponding to each of the multiple scanning threads or the CPU core number corresponding to the first business operation record, a first operation is performed to obtain the first queue.

4. The method according to claim 3, characterized in that The first operation includes at least one of the following: When the operation type of the first business operation record in each channel queue is consistent, the business operation record in each channel queue is sequentially added to a preset queue to obtain the first queue; In the case that there is at least one channel queue in which the operation type of the first business operation record is the target operation type, the first business operation record with the operation type of the target operation type is added to the preset queue to obtain the first queue.

5. The method according to claim 1, characterized in that The checking of the business operation records in the first queue and moving target business operation records that meet preset requirements to the second queue includes: assigning a corresponding weight to each business operation record in the first queue according to an operation type of each business operation record in the first queue; If a second business operation record is arranged after a third business operation record in the first queue and a weight of the second business operation record is less than or equal to a weight of the third business operation record, determining that the state of the second business operation record is a first state; wherein the second business operation record and the third business operation record are any two business operation records in the first queue, and processing of the second business operation record is suspended in the first state; If the fourth business operation record is arranged after the third business operation record in the first queue and the weight of the fourth business operation record is greater than the weight of the third business operation record, determining that the status of the fourth business operation record is the second status; If the fourth business operation record is in the second state, determining, based on an operation type of the fourth business operation record and a pre-acquired hash value, whether the fourth business operation record has been classified into a third queue, the third queue being used to store business operation records related to a second operation, the second operation including at least one of creation, deletion, rename, hard link, and soft link; If the fourth business operation record is not included in the third queue, the fourth business operation record is moved to the second queue.

6. The method according to claim 5, characterized in that The method further comprises: If the fourth business operation record has been included in the third queue, determining whether a fifth business operation record exists in the third queue based on a hash value of the fourth business operation record; wherein the name of the fifth business operation record is the same as the name of the fourth business operation record; If the fifth business operation record exists in the third queue and synchronization processing of the fifth business operation record has not been completed, move the fourth business operation record to the second queue; When the fifth business operation record does not exist in the third queue, the fourth business operation record is moved to the second queue.

7. The method according to claim 1, characterized in that The sending the business operation record in the second queue to the target cluster includes: Based on the multi-channel network constructed between the source cluster and the target cluster, concurrently sending the business operation records in the second queue to the target cluster; The method further comprises: When a synchronization success response message sent by the target cluster is received, the business operation record in the second queue is removed, and the business operation record stored on the disk is cleared.

8. The method according to claim 7, characterized in that The concurrently sending the business operation records in the second queue to the target cluster based on the multi-channel network constructed between the source cluster and the target cluster includes at least one of the following: Polling and querying each connection channel in the multi-channel network, and calling each connection channel in the multi-channel network in turn to send the business operation records in the second queue to the target cluster; In the event that there is task blocking in the multi-channel network, the business operation records in the second queue are sent to the target cluster through a target connection channel in the multi-channel network; wherein the target connection channel is a connection channel with the least number of task blocking in the multi-channel network.

9. A data synchronization device, characterized in that: Applied to a Redis server, the device includes: A record reading module is used to read at least two business operation records pre-solidified on the disk to build a first queue; an inspection and movement module, configured to inspect the business operation records in the first queue and move target business operation records that meet preset requirements to the second queue, wherein the target business operation record is at least one business operation record of the at least two business operation records; A data synchronization module is used to send the business operation records in the second queue to the target cluster.

10. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data synchronization method according to any one of claims 1 to 8.

11. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the data synchronization method according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the data synchronization method according to any one of claims 1 to 8.

Images