Data synchronization method, distributed storage cluster system and computer program product

Abstract

The invention discloses a data synchronization method, a distributed storage cluster system and a computer program product, and relates to the technical field of data storage. The sending sequence is set for the data cache queue corresponding to the classified to-be-synchronized data according to the target sending priority corresponding to the data type, and then the data in the data cache queue is sequentially sent to the slave server according to the sending sequence, so that classified transmission of the data can be realized, and key data can be preferentially transmitted; according to the data synchronization method and device, the data synchronization process is optimized, the technical problem that the data synchronization efficiency in the distributed storage cluster system is not high is solved, the data synchronization efficiency is improved, and smoother use experience is provided for users.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data synchronization method, a distributed storage cluster system, and a computer program product. Background Art

[0002] In distributed storage cluster systems, data on a master server is typically replicated to slave servers synchronously, enabling real-time data access from the slave servers. However, in related art, the master server sends a mix of different types of data to the slave servers, preventing them from timely accessing the required data. Furthermore, as data volume increases, the latency required for data synchronization increases, while also consuming more bandwidth and increasing server storage pressure, ultimately reducing data synchronization efficiency. Consequently, related art suffers from the technical problem of low data synchronization efficiency in distributed storage cluster systems.

[0003] With regard to related technologies, no effective solution has been proposed so far to the technical problem of low data synchronization efficiency in distributed storage cluster systems.

[0004] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the Invention

[0005] The present application provides a data synchronization method, a distributed storage cluster system, and a computer program product to at least solve the technical problem of low data synchronization efficiency in a distributed storage cluster system in the related art.

[0006] The present application provides a data synchronization method, wherein the distributed storage cluster system includes at least a master server and a slave server, and a data synchronization channel has been established between a master database on the master server and a slave database on the slave server, comprising: determining, from the master database, change data when a log of the master database changes, classifying the change data according to data type to obtain different groups of data to be synchronized, each group of data to be synchronized corresponding to a data type; setting a sending order for data cache queues of the different groups of data to be synchronized according to a target sending priority corresponding to the data type; receiving a data synchronization request from the slave server, and sending the data in the data cache queues of the different groups of data to be synchronized to the slave server through the data synchronization channel according to the sending order, so as to synchronize the data in the master database to the slave database.

[0007] In an exemplary embodiment, before setting the sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type, the method also includes: parsing the data request from the interactive interface to obtain a data list corresponding to the data request; determining a data interface for receiving the data request, and establishing a correspondence between the data interface and the data type corresponding to the data list; determining the reading frequency of the data interface based on the number of accesses to the data interface within a monitoring time period, and determining the reading frequency as the reading frequency of the data list based on the correspondence; and setting the sending priority for the data type according to the first sorting result of the reading frequency.

[0008] In an exemplary embodiment, setting a sending priority for the data type according to a first sorting result of the reading frequency includes: obtaining a first data list corresponding to the maximum reading frequency from the first sorting result, and setting a first sending priority for the first data list; determining a first data type of the first data list, and setting the first sending priority as a target sending priority for the first data type.

[0009] In an exemplary embodiment, setting a sending priority for the data type according to a first sorting result of the read frequency includes: obtaining a second data list corresponding to other read frequencies except the maximum read frequency from the first sorting result, and setting a second sending priority for the second data list, wherein the level of the second sending priority is lower than the level of the first sending priority; determining a second data type of the second data interface; determining a second data type of the second data list, and setting the second sending priority as a target sending priority for the second data type.

[0010] In an exemplary embodiment, a sending order is set for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type, including: sorting the target sending priority in order from high to low to obtain a second sorting result; determining the sorting order of the different groups of data to be synchronized according to the second sorting result; allocating a group of data cache queues to the same group of data to be synchronized, the group of data cache queues including at least one data cache queue, wherein the group of data cache queues has the same sending priority; setting a sending order for multiple groups of data cache queues of the different groups of data to be synchronized according to the sorting order.

[0011] In an exemplary embodiment, data in multiple data cache queues are sent to the slave server through the data synchronization channel in the sending order to synchronize the data in the master database to the slave database, including: determining the current group of data cache queues read in the sending order, the sending priority corresponding to the current group of data cache queues being higher than the sending priority of the remaining group of data cache queues that have not yet been read; sending the data of the current group of data cache queues to the slave server through the data synchronization channel; when it is determined that the data of the current group of data cache queues has been sent, clearing the data in the current group of data cache queues; when it is determined that the data of the current group of data cache queues has not been sent, determining the remaining data of the current group of data cache queues, and using the idle data cache queue of the current group of data cache queues to send the remaining data, the idle data cache queue is the queue in the current group of data cache queues other than the first data cache queue and the second data cache queue, wherein the first data cache queue is the queue that has sent data, and the second data cache queue is the queue that is sending data.

[0012] In an exemplary embodiment, before receiving the data synchronization request from the slave server, the method further includes: obtaining a synchronization notification message generated when a log of the master database changes, wherein the synchronization notification message is used to notify the slave server to generate a data synchronization request; and sending the synchronization notification message to the slave server.

[0013] The present application also provides a distributed storage cluster system, including: a master server and a slave server, the master server is used to determine the change data when the log of the master database changes from the master database, classify the change data according to the data type, and obtain different groups of data to be synchronized, each group of data to be synchronized corresponds to a data type; set a sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type; receive a data synchronization request from the slave server, and send the data in the data cache queues of the different groups of data to be synchronized to the slave server through the data synchronization channel according to the sending order, so as to synchronize the data in the master database to the slave database.

[0014] In an exemplary embodiment, the slave server is used to receive a synchronization notification message sent by the master server, determine the number of requests for the data synchronization request and the amount of data requested in a single request based on the disk performance parameters of the slave server, the network performance parameters of the slave server, and historical request records, and send the data synchronization request generated based on the number of requests and the amount of data requested in a single request to the master server.

[0015] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data synchronization methods when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data synchronization methods are implemented.

[0017] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data synchronization methods when executed by a processor.

[0018] Through this application, by classifying the changed data of the main server according to data type, setting the sending order for the data cache queue corresponding to the classified data to be synchronized according to the target sending priority corresponding to the data type, and then sending the data in the data cache queue to the slave server in sequence according to the sending order, not only can the classified transmission of data be achieved, but also the transmission of key data can be given priority, thereby optimizing the data synchronization process, solving the technical problem of low data synchronization efficiency in distributed storage cluster systems, improving the efficiency of data synchronization, and providing users with a smoother user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a data synchronization method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram illustrating the relationship between a data interface and a data list provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a priority sorting method provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a queue data transmission method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of another data synchronization method provided in an embodiment of the present application;

[0026] Figure 7 An architectural diagram of a distributed storage cluster system provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0030] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0031] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data synchronization method depends, the specific application environment architecture or specific hardware architecture is described here.

[0032] A data synchronization method provided in an embodiment of the present application is applied to a distributed storage cluster system, which includes at least a master server and a slave server, and a data synchronization channel has been established between a master database on the master server and a slave database on the slave server.

[0033] like Figure 1 As shown, this embodiment provides a data synchronization method, including:

[0034] S102, determining, from the master database, change data when a log of the master database is changed, and classifying the change data according to data type to obtain different groups of data to be synchronized, each group of data to be synchronized corresponding to a data type;

[0035] In this step, when the master server detects that a log has changed, it obtains the change data corresponding to the change log from the master database.

[0036] S104, setting a sending order for the data cache queues of the different groups of to-be-synchronized data according to the target sending priority corresponding to the data type;

[0037] S106, receiving a data synchronization request from the slave server, and sending the data in the data cache queue of the different groups of to-be-synchronized data to the slave server through the data synchronization channel in the sending order, so as to synchronize the data in the master database to the slave database.

[0038] The data synchronization request is sent by the slave server based on the synchronization notification message sent by the master server. The master server generates the synchronization notification message to the slave server when the log of the master database is changed.

[0039] Through this application, by classifying the changed data of the main server according to data type, setting the sending order for the data cache queue corresponding to the classified data to be synchronized according to the target sending priority corresponding to the data type, and then sending the data in the data cache queue to the slave server in sequence according to the sending order, not only can the classified transmission of data be achieved, but also the transmission of key data can be given priority, thereby optimizing the data synchronization process, solving the technical problem of low data synchronization efficiency in distributed storage cluster systems, improving the efficiency of data synchronization, and providing users with a smoother user experience.

[0040] In an exemplary embodiment, before setting the sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type, the method further includes: parsing the data request from the interactive interface to obtain a data list corresponding to the data request; determining a data interface for receiving the data request, and establishing a correspondence between the data interface and the data type corresponding to the data list; determining the read frequency of the data interface according to the number of accesses to the data interface within a monitoring time period, and determining the read frequency as the read frequency of the data list according to the correspondence; and setting a sending priority for the data type according to the first sorting result of the read frequency. In this embodiment, by setting the sending priority, it is convenient to synchronize frequently accessed data in the future. In addition, by determining the read frequency of the data interface in real time, the priority of the data type in data synchronization can be dynamically adjusted. For example, if it is found that the user frequently accesses data interface A, a higher sending priority is set for the data type corresponding to data interface A, thereby improving the pertinence of data synchronization according to the actual needs of the user.

[0041] In an exemplary embodiment, the process of setting a transmission priority for the data type according to the first sorting result of the read frequency includes: obtaining a first data list corresponding to the maximum read frequency from the first sorting result, setting a first transmission priority for the first data list; determining the first data type of the first data list, and setting the first transmission priority as the target transmission priority of the first data type. This embodiment achieves priority synchronization of the data most frequently accessed by users by setting the data type with the highest read frequency as the highest transmission priority. For example, alarm data in the master database is updated most frequently, and the data list corresponding to the alarm data is set to the highest transmission priority, giving priority to synchronizing the alarm data, thereby obtaining the latest and most accurate alarm data in real time from the slave database.

[0042] In an exemplary embodiment, the process of setting a sending priority for the data type according to the first sorting result of the reading frequency also includes: obtaining a second data list corresponding to other reading frequencies except the maximum reading frequency from the first sorting result, and setting a second sending priority for the second data list, wherein the level of the second sending priority is lower than the level of the first sending priority; determining the second data type of the second data interface; determining the second data type of the second data list, and setting the second sending priority to the target sending priority of the second data type. This embodiment reasonably allocates data synchronization resources by distinguishing the reading frequencies of different data types. For example, audit log data with a higher reading frequency is set to a higher sending priority, while performance data with a lower reading frequency is set to a lower sending priority. This ensures timely synchronization of high-frequency data and avoids low-frequency data from occupying too many synchronization resources, thereby achieving refined management of data synchronization.

[0043] In an exemplary embodiment, setting a sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type specifically includes the following steps: sorting the target sending priority in descending order to obtain a second sorting result; determining the sorting order of the different groups of data to be synchronized according to the second sorting result; allocating a group of data cache queues to the same group of data to be synchronized, the group of data cache queues including at least one data cache queue, wherein the group of data cache queues has the same sending priority; and setting a sending order for the multiple groups of data cache queues of the different groups of data to be synchronized according to the sorting order. In this embodiment, by determining the sending order of the cache queues according to the priority, orderly data synchronization is achieved. For example, the cache queue of alarm data is set to the highest priority in descending order, followed by the cache queue of audit log data, and finally the cache queue of performance data. In this way, critical alarm data can be synchronized first. At the same time, through the management of the cache queues, confusion during data synchronization is reduced, and the stability and efficiency of data synchronization are improved.

[0044] Optionally, in this embodiment, the data cache queue (hereinafter referred to as cache docking) can be initialized first. Specifically, the number of data rows required to store data is set for each data cache queue according to the sending priority. The higher the sending priority, the fewer the number of data rows.

[0045] In an exemplary embodiment, a process for transmitting data in multiple data cache queues to the slave server via the data synchronization channel in a transmission order to synchronize data in the master database to the slave database includes: determining a current group of data cache queues read in the transmission order, wherein the transmission priority corresponding to the current group of data cache queues is higher than the transmission priority of the remaining groups of data cache queues that have not yet been read; transmitting the data of the current group of data cache queues to the slave server via the data synchronization channel; if it is determined that the data of the current group of data cache queues has been transmitted, clearing the data in the current group of data cache queues; if it is determined that the data of the current group of data cache queues has not been transmitted, determining the remaining data of the current group of data cache queues and transmitting the remaining data using an idle data cache queue of the current group of data cache queues, wherein the idle data cache queues are queues in the current group of data cache queues other than a first data cache queue and a second data cache queue, wherein the first data cache queue is a queue that has transmitted data, and the second data cache queue is a queue that is currently transmitting data. This embodiment improves the continuity and integrity of data synchronization by processing the data cache queues one by one, and solves the problem of data inconsistency that may be caused by data synchronization interruption.

[0046] In an exemplary embodiment, before receiving a data synchronization request from the slave server, a synchronization notification message generated when a change occurs in the log of the master database can also be obtained, wherein the synchronization notification message is used to notify the slave server to generate a data synchronization request; and the synchronization notification message is sent to the slave server. This embodiment sends synchronization notification messages instantly so that the slave server can quickly respond to changes in the master database and initiate data synchronization requests in a timely manner. For example, when a new alarm occurs in the master database, a synchronization notification message is immediately generated and sent to the slave server. After receiving the message, the slave server immediately initiates a synchronization request for the alarm data. This ensures that the data on the slave server is consistent with the master server, thereby improving the real-time and accuracy of data synchronization.

[0047] Further, combined Figure 2 Explain the process of data synchronization. Figure 2 As shown, on the main server (corresponding to Figure 2 The master node of the server contains the virtual function modules "priority division module" and "sending module", and the slave server (corresponding to Figure 2 The slave node) includes a virtual function module "receiving module." It should be noted that the functions implemented by the virtual function module are actually implemented by the master server or the slave server. This application only uses the execution process of the virtual function module as an example to improve the readability of the implementation scheme of this application, and does not require that the corresponding functions of the master server or the slave server must be implemented by the virtual function module.

[0048] In the master server, the prioritization module is used to prioritize the master database ( Figure 2 The master database (referred to as the master database) sorts the data in different data lists by their sending priority. The master database stores data types such as docking platform information, cluster performance data, audit logs, alarm information, audit logs, performance data, node hardware configuration information, threshold information, user permission information, and user configuration information. Each data type is stored in a data list.

[0049] The corresponding "restapi request" issued by the receiving page (data synchronization request) reads the data saved in each data list. The data list queried when the underlying logic of each restapi interface that receives the "restapi request" is different. By monitoring the "restapi request" in the recent period, the following is obtained: Figure 3 A dictionary of the association between the interface name of the RESTAPI interface and the data list.

[0050] For example, the priority division module monitors the "restapi request" in the last day, and obtains the frequency of reading each data list based on the dictionary of the relationship between the restapi interface and the data list. Then, the priority is set according to the reading frequency of the data list. The higher the reading frequency, the higher the importance, and the higher the priority of the data list. At the same time, by monitoring the real-time access times of each restapi interface (i.e. Figure 4 The priority of each data list can be adjusted dynamically. Figure 4 As shown, the request frequency for interface "api_1" is 1 / 24, so its priority is n. The request frequency f for each data list is proportional to the number of requests to the corresponding REST API interface, i.e., f = cn, where c represents a constant coefficient. If the request frequency of each REST API interface changes, the priority of each data list can be dynamically adjusted based on the actual business changes in the cluster over a period of time.

[0051] In the main server, the sending module adds a process of determining the sending order and sending data according to the sending order based on the basic process. Figure 6 As shown: The client initiates a request to write data to the master node, and the master node first writes the data to itself. After the data is written, the master node records a Bin-log (binary) log. At the same time, after the master node configures the master-slave architecture, it will create a log dump thread (such as the Log dump thread) that specifically monitors the Bin-log log. When the Logdump thread monitors the log changes, the master node will notify the slave node to pull the changed data. In this embodiment, Figure 2 The basic process shown is similar to Figure 6 Similar, no further description is given here.

[0052] Slave nodes use dedicated I / O threads to receive notifications from the master node and then initiate data synchronization requests to the master. After requesting certain data from the master, the slave node writes the data to the relay log. Slave nodes also have database worker threads (such as SQL threads) that monitor relay log changes. When these changes occur, the slave node synchronizes data. For example, when a log change occurs, such as a newly written relay log, the slave node reads the log record and writes the corresponding change data to the corresponding slave database (also known as the slave database).

[0053] Based on the above solution, when the main database changes, the sending module groups the changed data according to the data list name (indicating the data type) to obtain the data to be synchronized. Then, the priority division module is used to obtain the priority of each data list in the recent period. Finally, the sending order of the grouped data to be synchronized is adjusted according to the priority of the data list. The data list with high priority is sent first, and the data with low priority is sent later to ensure that important log records are synchronized first. Specifically, the data to be synchronized can be placed in the cache queue in sequence. In this way, when the slave node requests data, the master node can send the data in the cache queue in the order of sending, thereby improving the reliability and timeliness of the data.

[0054] Further, combined Figure 5 The following steps describe the process of sending data in the buffer queue.

[0055] 1. Categorize the monitored change data according to the data list. For example, divide the data monitored for 1 second into the alarm data table, the audit log table, and the performance data table. The alarm data table will contain 2000 alarm data items, the audit log table will contain 3000 audit log data items, and the performance data table will contain 4000 performance data items.

[0056] 2. Initialize the cache queue: Set the number of storage rows for each queue. For example, assign a smaller number of sent entries to higher-level queues. For example, if the alarm priority is 1, the corresponding first-level queue will have a number of 100 sent entries. If the audit log priority is slightly lower, 2, the corresponding first-level queue will have a number of 200 sent entries. If the performance data priority is 3, the corresponding first-level queue will have a number of 300 sent entries.

[0057] 3. Sort the priorities of the data lists obtained by the priority classification module, and add the data in the higher priority data lists to a group of high-priority cache queues to synchronize the high-priority data as soon as possible. Add the data in the alarm data table to a group of cache queues with the highest priority.

[0058] 4. Select data from the highest priority cache queue and send it. For example, send 1000 pieces of alarm data.

[0059] 5. If all the data in the alarm data table has been sent, the data sending is terminated, and the corresponding set of cache queues is removed or cleared.

[0060] 6. If the data in the alarm data table has not been sent out, but the data slices in the current queue have been exhausted, the next cache queue is selected from a group of cache queues to continue sending.

[0061] A data slice is a segment into which data is divided to optimize efficiency during data transmission. "Data slice exhaustion" means that the currently processed data slice has been completely sent or processed, but the data in the entire data list has not yet been fully synchronized.

[0062] For example, when the sending module processes a high-priority data list (such as alarm data), the data in this data list is cut according to a certain number of rows (for example, every 1,000 data as a data slice), and each data slice is placed in a cache queue, and then the data slice is taken out from the cache queue for sending.

[0063] 7. Check whether the buffer queues of each priority level have been sent. For example, if the first level queue has been sent, the second level queue will continue to be sent.

[0064] 8. Repeat steps 4-6 until all data has been sent.

[0065] Within the slave server, the receiving module can monitor the slave node's disk pressure and other disk performance parameters in real time, as well as the network performance parameters of the network where the slave node and the master node reside, and the historical request frequency records to obtain the current request frequency. For example, when disk pressure is low and network port traffic is low, the number of requests can be increased based on the historical number of requests, and the data volume can be increased based on the historical data volume of a single request. Alternatively, when disk pressure is high and network port traffic is high, the number of requests and the data volume of a single request can be reduced accordingly based on the historical number of requests and historical data volume, avoiding wasted resources and reducing the impact on the cluster's services.

[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0067] like Figure 7 As shown, an embodiment of the present application further provides a distributed storage cluster system, including: a master server 72 and a slave server 74.

[0068] The master server is used to determine the change data when the log of the master database changes from the master database, classify the change data according to the data type, and obtain different groups of data to be synchronized, each group of data to be synchronized corresponds to a data type; set a sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type; receive the data synchronization request from the slave server, and send the data in the data cache queues of the different groups of data to be synchronized to the slave server through the data synchronization channel according to the sending order, so as to synchronize the data in the master database to the slave database.

[0069] Optionally, in a distributed storage cluster system, the master server serves as a control node for managing and allocating resources, and the slave server serves as a storage node for performing data read and write operations.

[0070] Through this application, by classifying the changed data of the main server according to data type, setting the sending order for the data cache queue corresponding to the classified data to be synchronized according to the target sending priority corresponding to the data type, and then sending the data in the data cache queue to the slave server in sequence according to the sending order, not only can the classified transmission of data be achieved, but also the transmission of key data can be given priority, thereby optimizing the data synchronization process, solving the technical problem of low data synchronization efficiency in distributed storage cluster systems, improving the efficiency of data synchronization, and providing users with a smoother user experience.

[0071] In an exemplary embodiment, the slave server is used to receive synchronization notification messages sent by the master server, determine the number of requests for the data synchronization request and the amount of data per request for the data synchronization request based on the slave server's disk performance parameters, the slave server's network performance parameters, and historical request records, and send the data synchronization request generated based on the number of requests and the amount of data per request to the master server. In this embodiment, the slave server dynamically adjusts the data synchronization request by comprehensively considering its own hardware performance and historical request conditions. If the slave server is located in an area with poor network conditions, it can reduce the number of network transmissions by reducing the number of requests and increasing the amount of data per request, thereby improving data synchronization efficiency.

[0072] In an exemplary embodiment, the main server is also used to parse the data request from the interactive interface before setting the sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type, to obtain a data list corresponding to the data request; determine the data interface for receiving the data request, and establish a correspondence between the data interface and the data type corresponding to the data list; determine the reading frequency of the data interface according to the number of accesses to the data interface within the monitoring time period, and determine the reading frequency as the reading frequency of the data list according to the corresponding relationship; set the sending priority for the data type according to the first sorting result of the reading frequency.

[0073] In an exemplary embodiment, the main server is also used to: obtain a first data list corresponding to the maximum reading frequency from the first sorting result, and set a first sending priority for the first data list; determine a first data type of the first data list, and set the first sending priority as a target sending priority for the first data type.

[0074] In an exemplary embodiment, the main server is also used to: obtain a second data list corresponding to other reading frequencies except the maximum reading frequency from the first sorting result, set a second sending priority for the second data list, wherein the level of the second sending priority is lower than the level of the first sending priority; determine the second data type of the second data interface; determine the second data type of the second data list, and set the second sending priority as the target sending priority of the second data type.

[0075] In an exemplary embodiment, the main server is also used to: sort the target sending priority in descending order to obtain a second sorting result; determine the sorting order of the different groups of data to be synchronized based on the second sorting result; allocate a group of data cache queues to the same group of data to be synchronized, the group of data cache queues including at least one data cache queue, wherein the group of data cache queues has the same sending priority; set the sending order for multiple groups of data cache queues of the different groups of data to be synchronized according to the sorting order.

[0076] In an exemplary embodiment, the main server is also used to: determine the current group data cache queue read in the sending order, the sending priority corresponding to the current group data cache queue is higher than the sending priority of the remaining group data cache queues that have not been read; send the data of the current group data cache queue to the slave server through the data synchronization channel; when it is determined that the data of the current group data cache queue has been sent, clear the data in the current group data cache queue; when it is determined that the data of the current group data cache queue has not been sent, determine the remaining data of the current group data cache queue, and use the idle data cache queue of the current group data cache queue to send the remaining data, the idle data cache queue is the queue in the current group data cache queue except the first data cache queue and the second data cache queue, wherein the first data cache queue is the queue that has sent data, and the second data cache queue is the queue that is sending data.

[0077] In an exemplary embodiment, the master server is also used to obtain a synchronization notification message generated when a log of the master database changes before receiving a data synchronization request from the slave server, wherein the synchronization notification message is used to notify the slave server to generate a data synchronization request; and send the synchronization notification message to the slave server.

[0078] For descriptions of features in the embodiments corresponding to the distributed storage cluster system, reference may be made to the relevant descriptions of the embodiments corresponding to the data synchronization method, which will not be described in detail here.

[0079] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data synchronization method embodiments.

[0080] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data synchronization method embodiments when running.

[0081] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0082] The embodiment of the present application also provides an electronic device, such as Figure 8As shown, the electronic device includes a memory 802 and a processor 804. The memory 802 stores a computer program, and the processor 804 is configured to execute the steps in any of the above method embodiments through the computer program.

[0083] Optionally, in this embodiment, the processor 804 may be configured to execute the following steps through a computer program:

[0084] S1, determining, from the master database, change data when a log of the master database is changed, and classifying the change data according to data type to obtain different groups of data to be synchronized, each group of data to be synchronized corresponding to a data type;

[0085] S2, setting a sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type;

[0086] S3, receiving a data synchronization request from the slave server, and sending the data in the data cache queue of the different groups of to-be-synchronized data to the slave server through the data synchronization channel in the sending order, so as to synchronize the data in the master database to the slave database.

[0087] Alternatively, those skilled in the art will appreciate that Figure 8 The structure shown is for illustration only. Figure 8 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 8 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 8 Different configurations shown.

[0088] Among them, the memory 802 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data synchronization method in the embodiment of the present application. The processor 804 executes various functional applications and data processing by running the software programs and modules stored in the memory 802, that is, realizing the above-mentioned data synchronization method. The memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 802 may further include a memory remotely located relative to the processor 804, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 802 can be used specifically, but not limited to, to store information such as distributed storage cluster system configuration files. As an example, such as Figure 8As shown, the memory 802 may include, but is not limited to, all modules in the regression testing apparatus described above. In addition, it may also include, but is not limited to, other module units in the regression testing apparatus described above, which will not be described in detail in this example.

[0089] Optionally, the transmission device 806 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 806 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 806 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0090] In addition, the electronic device further includes: a display 808; and a connection bus 810 for connecting various module components in the electronic device.

[0091] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data synchronization method embodiments are implemented.

[0092] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data synchronization method embodiments are implemented.

[0093] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0094] The above is a detailed introduction to a data synchronization method, a distributed storage cluster system, and a computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data synchronization method, characterized in that: Applied to a distributed storage cluster system, the distributed storage cluster system includes at least a master server and a slave server, a data synchronization channel has been established between a master database on the master server and a slave database on the slave server, including: Determining, from the master database, change data when a log of the master database is changed, and classifying the change data according to data type to obtain different groups of data to be synchronized, each group of data to be synchronized corresponding to a data type; Setting a sending order for the data cache queues of the different groups of to-be-synchronized data according to the target sending priority corresponding to the data type; Receive a data synchronization request from the slave server, and send the data in the data cache queue of the different groups of to-be-synchronized data to the slave server through the data synchronization channel in the sending order, so as to synchronize the data in the master database to the slave database.

2. The data synchronization method according to claim 1, characterized in that: Before setting a sending order for the data cache queues of the different groups of to-be-synchronized data according to the target sending priorities corresponding to the data types, the method further includes: Parsing the data request from the interactive interface to obtain a data list corresponding to the data request; Determine a data interface for receiving the data request, and establish a correspondence between the data interface and the data type corresponding to the data list; determining a reading frequency of the data interface according to the number of accesses to the data interface within a monitoring period, and determining the reading frequency as a reading frequency of the data list according to the corresponding relationship; A sending priority is set for the data type according to the first sorting result of the reading frequency.

3. The data synchronization method according to claim 2, wherein: Setting a sending priority for the data type according to the first sorting result of the reading frequency includes: Obtaining a first data list corresponding to a maximum read frequency from the first sorting result, and setting a first sending priority for the first data list; A first data type of the first data list is determined, and the first sending priority is set as a target sending priority of the first data type.

4. The data synchronization method according to claim 3, characterized in that: Setting a sending priority for the data type according to the first sorting result of the reading frequency includes: Obtaining a second data list corresponding to other reading frequencies except the maximum reading frequency from the first sorting result, and setting a second sending priority for the second data list, wherein the second sending priority is lower than the first sending priority; determining a second data type of the second data interface; A second data type of the second data list is determined, and the second transmission priority is set as a target transmission priority of the second data type.

5. The data synchronization method according to claim 1, wherein: Setting a sending order for the data cache queues of the different groups of to-be-synchronized data according to the target sending priority corresponding to the data type includes: Sorting the target sending priorities in descending order to obtain a second sorting result; Determining a sorting order of the to-be-synchronized data of the different groups according to the second sorting result; Allocating a group of data cache queues for the same group of data to be synchronized, the group of data cache queues including at least one data cache queue, wherein the group of data cache queues have the same sending priority; A sending order is set for the multiple groups of data cache queues of the different groups of data to be synchronized according to the sorting order.

6. The data synchronization method according to claim 1, characterized in that: Sending the data in the data cache queues of the different groups of to-be-synchronized data to the slave server through the data synchronization channel in the sending order, so as to synchronize the data in the master database to the slave database, includes: Determining a current group of data cache queues read in the sending order, wherein a sending priority corresponding to the current group of data cache queues is higher than a sending priority of remaining group of data cache queues that have not yet been read; Sending the data of the current group data cache queue to the slave server through the data synchronization channel; When it is determined that the data in the current group data cache queue has been sent, clearing the data in the current group data cache queue; When it is determined that the data of the current group data cache queue has not been sent, the remaining data of the current group data cache queue is determined, and the idle data cache queue of the current group data cache queue is used to send the remaining data, and the idle data cache queue is a queue in the current group data cache queue except the first data cache queue and the second data cache queue, wherein the first data cache queue is a queue that has sent data, and the second data cache queue is a queue that is sending data.

7. The data synchronization method according to claim 1, characterized in that: Before receiving the data synchronization request from the slave server, the method further includes: Acquire a synchronization notification message generated when a log of the master database is changed, wherein the synchronization notification message is used to notify the slave server to generate a data synchronization request; The synchronization notification message is sent to the slave server.

8. A distributed storage cluster system, characterized in that: include: A master server and a slave server, the master server is used to determine the change data when the log of the master database changes from the master database, classify the change data according to the data type, and obtain different groups of data to be synchronized, each group of data to be synchronized corresponds to a data type; set a sending order for the data cache queues of the different groups of data to be synchronized according to the target sending priority corresponding to the data type; receive a data synchronization request from the slave server, and send the data in the data cache queues of the different groups of data to be synchronized to the slave server through the data synchronization channel according to the sending order, so as to synchronize the data in the master database to the slave database.

9. The distributed storage cluster system according to claim 8, characterized in that: The slave server is used to receive the synchronization notification message sent by the master server, determine the number of requests for the data synchronization request and the amount of data requested in a single request based on the disk performance parameters of the slave server, the network performance parameters of the slave server and the historical request records, and send the data synchronization request generated based on the number of requests and the amount of data requested in a single request to the master server.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 7 are implemented.

Images