A message queue-based data synchronization method, device and medium

Abstract

This application discloses a data synchronization method, device, and medium based on a message queue. The method includes: determining the concurrency of business data and the number of category lists for the business data according to business requirements; creating all category lists in the message queue; adding a send lock to each category list; creating a buffer corresponding to each category list; obtaining the changed dataset; encapsulating the changed dataset into a message; adding a sequence attribute and the corresponding category list to the message; acquiring the send lock; adding the message to the corresponding buffer; adding a message lock to the buffer and executing the operation; sending the message to the corresponding category list in the message queue; and releasing the send lock and message lock after the message is sent. By setting multiple topics for each type of business data in the message queue, the message ordering between the sender and consumer is guaranteed. Batch message sending is achieved through setting buffers corresponding to topics and the mechanism of setting send locks and buffers.

Description

Technical Field

[0001] This application relates to the field of data management, specifically to a data synchronization method, device, and medium based on message queues. Background Technology

[0002] With the development of the modern information environment, more and more enterprises need to synchronize data from different systems or data sources. These data sources may be generated by different platforms and databases in different regions and time periods, and there are differences between the data sources, such as data format, data type, and data structure. In order to ensure the accuracy and consistency of the data, it is necessary to synchronize these data.

[0003] The common practice for data synchronization is to use Change Data Capture (CDC) technology to periodically or near-real-time pull changed data from the database and synchronize the captured data to other databases. However, some databases do not provide an interface for CDC, making data synchronization impossible. In such cases, data synchronization can only be achieved by intercepting data changes at the application layer, collecting the changed dataset, and sending the dataset to a message queue. This method involves multiple sending steps, such as sending data to a message queue for temporary storage before synchronizing it to other databases. If not handled properly, this can easily lead to significant performance delays and data inconsistencies. Summary of the Invention

[0004] To address the aforementioned issues, this application proposes an SDN-based switch command-line annotation method, including:

[0005] Based on business requirements, determine the concurrency level of the business data, and based on the concurrency level, determine the number of category lists for the business data;

[0006] Create all the aforementioned category lists in the message queue, add a send lock to each of the aforementioned category lists, and create a buffer corresponding to each of the aforementioned category lists;

[0007] Obtain the changed dataset, encapsulate the changed dataset into a message, and add a sequence attribute to the message;

[0008] Based on the sequence attribute, the corresponding category list is determined, thereby determining the corresponding buffer, acquiring the send lock, and adding the message to the corresponding buffer;

[0009] Add a message lock to the buffer and execute the operation to send the message from the buffer to the corresponding category list in the message queue. After the message is sent, release the send lock and the message lock.

[0010] On the other hand, this application also proposes a message queue-based data synchronization device, comprising:

[0011] At least one processor; and,

[0012] A memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform a message queue-based data synchronization method as described in the above example.

[0014] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as the message queue-based data synchronization method described in the above example.

[0015] The data synchronization method based on message queues proposed in this application can bring the following benefits:

[0016] By setting multiple topics in the message queue for each type of business data, the problem of large performance latency caused by high concurrency of business data can be solved, the message ordering between the sender and consumer can be guaranteed, and the performance and throughput of message sending can be improved.

[0017] By setting up a buffer corresponding to the topic and a mechanism for setting up send locks and buffers, batch sending of messages is achieved, preventing message inconsistencies caused by new messages during message sending. Furthermore, the batch size and timing of message sending can be automatically controlled without the need for additional threads, simplifying message synchronization management. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart illustrating a data synchronization method based on a message queue, as described in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram illustrating the implementation process of a data synchronization method based on a message queue, as described in this application.

[0021] Figure 3 This is a schematic diagram of a message queue-based data synchronization device in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0024] like Figure 1 As shown in the figure, this application provides a data synchronization method based on a message queue, including:

[0025] S101: Determine the concurrency level of the business data based on business requirements, and determine the number of category lists for the business data based on the concurrency level.

[0026] Data synchronization refers to the process of updating data in multiple data sources in a consistent manner to ensure the accuracy and integrity of the data. The traditional method of data synchronization is to use CDC data synchronization technology to capture data changes in the database in real time or near real time, parse the changed data, and synchronize the changed data to other database storage, thereby maintaining data consistency.

[0027] Meanwhile, some small databases do not provide a CDC interface and cannot achieve data synchronization through CDC. In this case, interception can be performed at the application layer to collect data changes. Within a transaction, the collected change dataset is sent to a message queue for temporary storage. The consumer obtains the change dataset from the message queue and consumes the change dataset, thus obtaining the changed data and synchronizing the changed data to other databases, thereby ensuring data consistency.

[0028] However, the data synchronization method using message queues can lead to significant performance delays and data inconsistencies due to the temporary storage of messages in the message queue, which increases the time it takes for messages to travel from the sender to the consumer. Therefore, this application's embodiments address these issues by implementing corresponding processing measures during the data synchronization process. These measures primarily involve the following two points: In the synchronous message sending mode, batch message sending is achieved through data buffering and locking mechanisms, with automatic control over batch size and sending timing, eliminating the need for additional thread management; to improve sending performance and throughput, even when message data is distributed to multiple category lists, the message order across these categories is guaranteed at both the sender and consumer ends.

[0029] Specifically, based on business requirements, the concurrency of the business data is determined, thereby determining the number of category lists. In typical data synchronization scenarios, only one category list needs to be created for each type of business data. In this embodiment, the category lists in the database are referred to as "topics," which are used to distinguish different types of data or messages. However, due to the complex business relationships of some enterprises or organizations, the corresponding business data concurrency is high, resulting in a large volume of messages. Using only one topic is insufficient to support the high concurrency of this business data, leading to high latency. Therefore, for this type of high-concurrency business data, multiple topics are created for temporary storage to ensure the message ordering between the sender and consumer.

[0030] S102: Create all the category lists in the message queue, add a send lock to each category list, and create a buffer corresponding to the category list.

[0031] Specifically, after determining the number of topics, all topics are created in the message queue and named. The naming rule can be business name_number to distinguish the data in the message queue. Each topic contains three dictionaries: a buffer dictionary, a buffer lock dictionary, and a buffer execution lock dictionary. The key in the buffer dictionary, buffer lock dictionary, and buffer execution lock dictionary is topicId. The buffer dictionary is mainly used to create buffers, the buffer lock dictionary is mainly used to control the thread safety of adding to the buffer, and the buffer execution lock dictionary value is mainly used to control the thread safety when sending from the buffer.

[0032] Furthermore, by adding a lock dictionary to the buffer, a send lock is added to each topic, and the ID of the corresponding buffer is determined based on the ID of the topic. Then, based on the ID of the buffer, the corresponding buffer is created through the buffer dictionary of the topic. The buffer is created to temporarily store the message array, and the messages sent to the topic are added to the buffer corresponding to the topic for temporary caching.

[0033] S103: Obtain the changed dataset, encapsulate the changed dataset into a message, and add the sequence attribute to the message.

[0034] Specifically, the interception is performed at the application layer to collect data changes and obtain the change dataset. The obtained change dataset is then encapsulated in various ways, such as encapsulating it into classes and objects, structures, arrays, etc. In this embodiment, an array is used as an example for explanation. The change dataset is organized in order and encapsulated into a message array.

[0035] Furthermore, a `sequence` attribute is added to the message array. This attribute identifies the order of messages in the message queue, ensuring that the order of messages in the queue matches the order in which they were sent. The `sequence` value for each message array is set incrementally to guarantee its uniqueness. Simultaneously, a maximum sequence number table is created in the business database to store the sequence values ​​corresponding to each message. The structure of the `sequence` table is shown in Table 1, and it includes two fields: `TopicId` and `MaxSequence`.

[0036] TopicId Varchar Queue number MaxSequence Long Current maximum serial number

[0037] Table 1

[0038] For example, such as Figure 2 As shown, sales order data is used as business data. Based on the concurrency of this sales order data, multiple topics are created and named as order_0, order_1...order_N, and placed in the order topic list of the message queue. Based on the ID of each topic, the ID of the corresponding buffer is determined, namely the first buffer, the second buffer...the Nth buffer. These buffers are placed in the sender's pending buffer to temporarily cache order messages generated in the order thread. The order thread includes M threads that generate multiple messages. A sequence attribute is added to each message array, and the sequence value is set incrementally, namely seq:11, seq12...seq1N. These messages are added to the corresponding buffers for temporary caching according to the preset routing rules.

[0039] Order messages whose sales order data is sent to the order topic list are first temporarily cached in the corresponding buffer in the pending-send buffer.

[0040] S104: Determine the corresponding category list based on the sequence attribute, thereby determining the corresponding buffer, acquiring the send lock, and adding the message to the corresponding buffer.

[0041] Specifically, after obtaining the changed dataset and encapsulating it into messages, the ID of the topic to which the message needs to be sent is obtained according to the sequence value corresponding to each message and through the preset routing rules. This yields the ID of the buffer corresponding to the topic. The preset routing rule is Formula 1: topicId = sequence % topicNum. In Formula 1, topicId is the ID of the corresponding topic, sequence is the sequence value of the message, and topicNum is the number of topics corresponding to the business data.

[0042] Furthermore, the send lock for the corresponding topic is acquired to ensure that other threads do not modify or delete the buffer before reading the corresponding buffer, thus preventing invalid or inconsistent data from being read. After acquiring the send lock, the existence of the buffer is determined based on the ID of the buffer corresponding to the acquired topic.

[0043] If the buffer does not exist, a new buffer corresponding to the topic is created to prevent the topic from having no corresponding buffer after the buffer is deleted after all messages in the buffer have been sent. After the new buffer is created, the corresponding message array is added to the new buffer. If the buffer exists based on the buffer ID, the corresponding message array is added directly to the buffer and the message array is temporarily cached in the buffer.

[0044] S105: Add a message lock to the buffer and execute the operation, send the message from the buffer to the corresponding category list in the message queue, and release the send lock and the message lock after the message is sent.

[0045] Before sending a message, a lock is added to the buffer to ensure that only one thread executes the sending of messages in the buffer. The buffer lock is also locked to prevent new messages from being added to the buffer. Furthermore, the message is sent from the buffer to the topic corresponding to the message queue. After the message sending in the buffer is completed, the execution result is returned and stored in the message sending result table in the business database. The table structure is shown in Table 2.

[0046] TopicId Varchar Queue number Sequence Long Current serial number

[0047] Table 2

[0048] Furthermore, the buffer is deleted to achieve buffer switching. During message sending, if a new message is generated that needs to be sent to the topic, a new buffer corresponding to the topic will be created, and the newly generated message will be automatically added to the new buffer. After the message is sent, the buffer is set to the completed state. If there is an exception, it is set to the sending failure state, and the sending lock corresponding to the topic is released. The next batch of threads will acquire the sending lock and trigger the sending of the next batch of messages, without the need for additional thread management.

[0049] like Figure 2 As shown, after the sender sends the message to the message queue, the consumer sorts the messages according to the SEQ, that is, according to the message sequence value. It listens to the topic in the consumption queue, retrieves the message from the specified topic according to the current offset, and consumes the message accordingly. Furthermore, based on the sequence value corresponding to the current topic, it obtains the ID of the topic corresponding to the next message using formula 2: topicID = (sequence + 1) % topicNum, and then retrieves the next message from the specified topic.

[0050] If the above method can retrieve the corresponding message, the above steps are repeated; if it cannot be retrieved, the corresponding message can be retrieved by querying the business database based on the topic ID or sequence value.

[0051] like Figure 3 As shown in the embodiments of this application, a data synchronization device based on a message queue is also proposed, including:

[0052] At least one processor; and,

[0053] A memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a message queue-based data synchronization method as described in any of the above embodiments.

[0055] This application also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as the message queue-based data synchronization method described in any of the above embodiments.

[0056] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0057] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data synchronization method based on a message queue, characterized in that, include: Based on business requirements, determine the concurrency level of the business data, and based on the concurrency level, determine the number of category lists for the business data; Create all the aforementioned category lists in the message queue, add a send lock to each category list, and create a buffer corresponding to each category list, specifically including: Add a lock dictionary to the buffer in the category list, and add a send lock to each category list; Based on the ID of the category list, determine the ID of the corresponding buffer. Based on the ID of the buffer, create the corresponding buffer through the buffer dictionary of the category list to temporarily store the message array. Obtain the changed dataset, encapsulate the changed dataset into a message, and add a sequence attribute to the message, specifically including: Obtain the change dataset from the business database, and encapsulate the change dataset into a message; Add a sequence attribute to the message and set the sequence value corresponding to the message in an incremental manner to ensure that the sequence value corresponding to the message is unique. Create a maximum sequence number table in the business database to store the sequence value corresponding to the message. Based on the sequence attribute, the corresponding category list is determined, thereby determining the corresponding buffer, acquiring the send lock, and adding the message to the corresponding buffer, specifically including: Based on the sequence value corresponding to the message, the ID of the category list is determined by the routing rules, thereby determining the ID of the corresponding buffer; Obtain the send lock corresponding to the category list, determine whether the buffer exists based on the ID of the buffer, and add the message to the corresponding buffer; Adding a message lock to the buffer and executing the command, sending the message from the buffer to the corresponding category list in the message queue, and releasing the send lock and the message lock after the message is sent, specifically includes: Add a message lock to the buffer and execute the message lock to prevent new messages from being added to the buffer; The message is sent from the buffer to the corresponding category list in the message queue. After the message in the buffer is sent, the sending and the message lock are released for the next sending of the message. And then delete the buffer to achieve the switching of the buffer; Also includes: If a new message is generated, a new buffer corresponding to the category list is created, and the new message is automatically added to the new buffer to enable the next sending of the new message.

2. The method according to claim 1, characterized in that, The step of obtaining the send lock corresponding to the category list, determining whether the buffer exists based on the buffer ID, and adding the message to the corresponding buffer specifically includes: Obtain the send lock corresponding to the category list, and determine whether the buffer exists based on the ID of the buffer; If the buffer does not exist, a new buffer is created, and the message is added to the corresponding buffer. If the buffer exists, the message is directly added to the corresponding buffer.

3. The method according to claim 1, characterized in that, After sending the message from the buffer to the corresponding topic in the message queue and releasing the send lock after the message sending is completed, the method further includes: After the message is sent to the message queue, the ID of the corresponding category list is obtained based on the sequence value; The messages in the corresponding category list in the message queue are sent to the consumer, and the consumer consumes the messages.

4. A data synchronization device based on a message queue, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform actions as described in any one of claims 1 to 3. Data synchronization methods based on message queues.

5. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured as follows: the data synchronization method based on message queues as described in any one of claims 1 to 3.

Images