A data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and storage medium, and belongs to the technical field of big data. The method comprises the following steps: setting an offset for each data change event occurring on each server and saving each data change event and the offset of each data change event into a data table; performing the following operation on each data change event in the order of the offset recorded in the data table: taking the data change event as a to-be-synchronized data change event, sending a data synchronization instruction for the to-be-synchronized data change event to part or all of the servers according to the offset of the to-be-synchronized data change event and the completed offset corresponding to each server recorded in the offset table, receiving the synchronization completion message sent by part or all of the servers and updating the completed offset of the corresponding server to the offset of the to-be-synchronized data change event based on the synchronization completion message, and realizing the high availability of the database.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the advent of the big data era and the continuous growth of data volume, the demand for data queries is becoming increasingly frequent. Users have increasingly strong requirements for data query speed and uninterrupted data query services, and do not want to experience long waiting times or see "query timeout" or "database busy" messages when accessing and querying data. A single database cannot support the corresponding rapid increase in storage and computing intensity caused by the rapid growth of data and query volume. In this situation, it is necessary to expand the existing database with hardware or directly replace it with a higher-performance database. However, no matter how powerful a single database server is, it cannot meet the ever-increasing demand for data volume and query volume, resulting in poor availability. Moreover, each expansion or replacement not only wastes hardware resources but also increases user costs.

[0003] Therefore, how to achieve high availability of databases is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and storage medium to achieve high availability of a database.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to a processing node in a distributed storage system. The distributed storage system includes the processing node and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship, each server providing read and write functionality to a user. The method includes:

[0006] Set an offset for each data change event that occurs on each server, and save each data change event and its offset to a data table;

[0007] For each data change event, perform the following operations according to the order of the offsets recorded in the data table:

[0008] The data change event is taken as the data change event to be synchronized, and according to the offset of the data change event to be synchronized and the completed offset of each server recorded in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers; the completed offset of any server refers to the offset of the data change event that the server has currently completed synchronizing.

[0009] Receive synchronization completion messages from some or all of the multiple servers, and update the completed offset of the corresponding server to the offset of the data change event to be synchronized based on the synchronization completion message.

[0010] In some embodiments, before setting the offset for each data change event occurring on each server, the method further includes:

[0011] Each server is sent a request to retrieve a log file; the log file request is used to instruct the corresponding server to send a log file to the processing node, and the log file is used to record data change events that occur on the server.

[0012] Based on the log files received from each server, obtain the data change events that occurred on each server.

[0013] In some embodiments, each data change event carries a timestamp indicating when the data change event occurred;

[0014] The step of setting offsets for data change events occurring on each server and saving each data change event and its offset to a data table includes:

[0015] Sort the data change events that occurred on multiple servers in ascending order of their timestamps;

[0016] Set a unique offset for each data change event based on the order in which they are arranged.

[0017] Based on the correspondence between each data change event and the server, the server identifier to which each data change event belongs is marked;

[0018] Save each data change event, the offset of each data change event, and the server identifier labeled for each data change event to the data table.

[0019] In some embodiments, before sorting the data change events occurring on multiple servers in ascending order of their timestamps, the method further includes:

[0020] Determine if there are at least two identical data change events that were generated by different servers;

[0021] If it is determined that there exists, the data change event with the largest timestamp among the at least two data change events is retained, and the remaining data change events among the at least two data change events are deleted.

[0022] In some embodiments, the synchronization completion message is sent by any server when it determines that the synchronization of the data change event to be synchronized is complete, or by any server when it determines that the server identifier of the data change event to be synchronized indicated in the data synchronization instruction is consistent with the identifier of that server.

[0023] In some embodiments, after saving each data change event and its offset to a data table, the method further includes:

[0024] When it is determined that the preset cleanup conditions are met, the minimum offset is determined based on the completed offset corresponding to each server recorded in the offset table.

[0025] Based on the minimum offset, offsets in the data table that are less than or equal to the minimum offset and their corresponding data change events are cleaned up.

[0026] In some embodiments, the method of receiving synchronization completion messages sent by some or all of the plurality of servers includes:

[0027] Within a preset time period, receive synchronization completion messages sent by some or all of the multiple servers;

[0028] If it is determined that no synchronization completion message is received from any server within the preset time, then the corresponding processing is performed according to the cause of the server failure.

[0029] In some embodiments, the method of performing corresponding processing based on the cause of the server failure includes:

[0030] If the server malfunction is due to the server being down, then once the server is detected to be running normally, the data synchronization command will be resent to the server.

[0031] If the server malfunctions because the server does not exist, then the server and its corresponding offset are deleted from the offset table.

[0032] Secondly, embodiments of this application provide a data processing apparatus applied to a processing node in a distributed storage system. The distributed storage system includes the processing node and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship, each server providing read and write functionality to users, including:

[0033] The configuration module is used to set offsets for each data change event that occurs on each server, and save each data change event and its offset to a data table;

[0034] The execution module is used to perform the following operations for each data change event, according to the offset order recorded in the data table:

[0035] The data change event is taken as the data change event to be synchronized, and according to the offset of the data change event to be synchronized and the completed offset of each server recorded in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers; the completed offset of any server refers to the offset of the data change event that the server has currently completed synchronizing.

[0036] The receiving module is used to receive synchronization completion messages sent by some or all of the multiple servers, and update the completed offset of the corresponding server to the offset of the data change event to be synchronized based on the synchronization completion message.

[0037] In some embodiments, it also includes:

[0038] The acquisition module is used to send a log file retrieval request to each server before setting the offset for each data change event that occurs on each server; the log file request is used to instruct the corresponding server to send the log file to the processing node, and the log file is used to record the data change events that occur on the server.

[0039] Based on the log files received from each server, obtain the data change events that occurred on each server.

[0040] In some embodiments, each data change event carries a timestamp indicating when the data change event occurred; the setting module is specifically used for:

[0041] Sort the data change events that occurred on multiple servers in ascending order of their timestamps;

[0042] Set a unique offset for each data change event based on the order in which they are arranged.

[0043] Based on the correspondence between each data change event and the server, the server identifier to which each data change event belongs is marked;

[0044] Save each data change event, the offset of each data change event, and the server identifier labeled for each data change event to the data table.

[0045] In some embodiments, before the setting module sorts the data change events occurring on multiple servers in ascending order of their timestamps, it further includes:

[0046] The judgment module is used to determine whether there are at least two identical data change events that were generated by different servers.

[0047] If it is determined that there exists, the data change event with the largest timestamp among the at least two data change events is retained, and the remaining data change events among the at least two data change events are deleted.

[0048] In some embodiments, the synchronization completion message is sent by any server when it determines that the synchronization of the data change event to be synchronized is complete, or by any server when it determines that the server identifier of the data change event to be synchronized indicated in the data synchronization instruction is consistent with the identifier of that server.

[0049] In some embodiments, after the setting module saves each data change event and the offset of each data change event to the data table, it further includes:

[0050] The cleanup module is used to determine the minimum offset based on the completed offset corresponding to each server recorded in the offset table when it is determined that the preset cleanup conditions are met.

[0051] Based on the minimum offset, offsets in the data table that are less than or equal to the minimum offset and their corresponding data change events are cleaned up.

[0052] In some embodiments, the receiving module is specifically used for:

[0053] Within a preset time period, receive synchronization completion messages sent by some or all of the multiple servers;

[0054] If it is determined that no synchronization completion message is received from any server within the preset time, then the corresponding processing is performed according to the cause of the server failure.

[0055] In some embodiments, the receiving module is specifically used for:

[0056] If the server malfunction is due to the server being down, then once the server is detected to be running normally, the data synchronization command will be resent to the server.

[0057] If the server malfunctions because the server does not exist, then the server and its corresponding offset are deleted from the offset table.

[0058] Thirdly, embodiments of this application provide an electronic device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor performs the above-described data processing method.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a processor, which, when run on the processor, causes the processor to perform the aforementioned data processing method.

[0060] In this embodiment, the processing node is applied to a distributed storage system. The distributed storage system includes processing nodes and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship. Each server provides read and write functions to users. An offset is set for each data change event occurring on each server, and each data change event and its offset are saved in a data table. According to the order of the offsets recorded in the data table, the following operations are performed on each data change event: the data change event is treated as a data change event to be synchronized, and based on the offset of the data change event to be synchronized and the completed offset recorded for each server in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers. The completed offset for any server indicates the data change that the server has currently completed synchronizing. The offset of this event is used to receive synchronization completion messages from some or all of multiple servers. Based on these messages, the completed offset of the corresponding server is updated to the offset of the data change event to be synchronized. In this way, multiple servers are in a master-slave relationship, and each server provides read and write functions for users, which can improve data query speed. Moreover, if any database server fails, the other master-slave database servers can still work normally, ensuring data security and achieving high availability of the database. At the same time, a data table and an offset table are set. When a data change event occurs on any database server, the offset of this data change event is set and stored in the data table. Based on the data change events in the data table, the database servers are synchronized and updated. The offset of any data change event that has been synchronized and completed on any database server is also updated, ensuring data consistency across database servers. Attached Figure Description

[0061] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0062] Figure 1This application provides an illustration of an application scenario for a data processing method.

[0063] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0064] Figure 3 A schematic diagram illustrating the framework of a data processing method provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the hardware structure of an electronic device for implementing a data processing method, provided as an embodiment of this application. Detailed Implementation

[0067] To achieve high availability of the database, embodiments of this application provide a data processing method, apparatus, electronic device, and storage medium.

[0068] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features described herein can be combined with each other without conflict. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0069] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0070] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, it can refer to a direct connection, an indirect connection through an intermediate medium, or a connection within two devices. Those skilled in the art can understand the specific meaning of the above term in this application based on the specific circumstances.

[0071] For ease of understanding, the technical terms used in this application are as follows:

[0072] MySQL is a relational database management system that speeds up data access and improves flexibility by storing data in different tables, rather than putting all the data in one large repository.

[0073] A timestamp is a complete and verifiable piece of data that indicates that a piece of data existed before a specific time. It is usually a sequence of characters used to uniquely identify a moment in time.

[0074] Data Definition Language (DDL) is a language used to define and manage all objects in a MySQL database. The main commands include CREATE, ALTER, and DROP. DDL is mainly used for initialization tasks such as defining or changing the structure, data types, links, and constraints between tables.

[0075] Data Manipulation Language (DML) is a language used to manipulate data in a database. It allows you to perform INSERT, DELETE, and UPDATE operations on data in a database table without changing the table structure.

[0076] Figure 1 This application scenario diagram illustrates a data processing method provided in an embodiment of the present application, including a processing node 100 and a database server cluster 200 in a distributed storage system.

[0077] Processing node 100 can process data from multiple servers in the database server cluster 200. It sets offsets for data change events occurring on each server and saves each data change event and its offset to a data table. This data table can be pre-set, located within the processing node, or within the database server cluster; this is not limited. Following the order of offsets recorded in the data table, for each data change event, the following operations are performed: the data change event is treated as a data change event to be synchronized; based on the offset of the data change event to be synchronized and the completed offset recorded for each server in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers; the completed offset for any server refers to the offset of the data change event that has already been synchronized on that server; synchronization completion messages are received from some or all of the multiple servers, and the completed offset of the corresponding server is updated to the offset of the data change event to be synchronized based on the synchronization completion messages.

[0078] The database server cluster consists of 200 servers with a master-slave relationship. Each server provides read and write functionality to users. The database can be a MySQL database.

[0079] Among them, the processing node 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center.

[0080] Specifically, the processing node 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130, and an output device 140, etc. The input device 130 may include a keyboard, mouse, touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0081] The memory 120 may include read-only memory (ROM) and random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In this embodiment, the memory 120 may be used to store a program for any data processing method in this embodiment.

[0082] The processor 110 executes the steps of any data processing method in the embodiments of this application according to the program instructions stored in the memory 120.

[0083] Processing node 100 and database server cluster 200 are connected via the Internet to enable communication between them. Optionally, the Internet uses standard communication technologies and / or protocols. The Internet is typically the Internet, but can also be any network, including but not limited to any combination of Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless networks, private networks or virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0084] It should be noted that in the various embodiments of this application, the data processing method is applied to Figure 1 The application architecture shown is used as an example for illustrative purposes.

[0085] After introducing the application scenarios of the embodiments of this application, the data processing proposed in this application will be described below with specific embodiments. Figure 2 A flowchart of a data processing method provided in an embodiment of this application, the method being applied to Figure 1 The method includes the following steps in the processing node.

[0086] In step 201, an offset is set for each data change event that occurs on each server, and each data change event and its offset are saved to a data table.

[0087] In practical implementation, the distributed storage system includes processing nodes and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship. Each server provides read and write functions to users. The database server cluster can be composed of multiple MySQL servers in a master-slave relationship. The master-slave relationship means that any copy on any server is a complete database. Each server can be used as a master server or as a slave server of any other server.

[0088] In some embodiments, a request to retrieve log files can be sent to each server separately. The log file request is used to instruct the corresponding server to send the log file to the processing node. The log file is used to record data change events that occur on the server. Then, based on the log files sent by each server, the data change events that occur on each server are obtained. Then, based on the data change events that occur on each server, an offset is set for each data change event that occurs on each server.

[0089] In practical implementation, taking MySQL as an example, the MySQL log file can be a binary log (Binlog file). Each MySQL Binlog file records data change events within that MySQL instance, such as DDL statements executed on the database and DML statements executed on the data. Based on the obtained Binlog file, the data change events that occurred on each server, as well as other key information about each data change event, such as the timestamp of the time when each data change event occurred, can be obtained.

[0090] In some embodiments, setting offsets for data change events occurring on each server and saving each data change event and its offset to a data table can be implemented as follows: First, sort the data change events occurring on multiple servers in ascending order of their timestamps. Then, set a unique offset for each data change event based on its sorting order. Next, label each data change event with its corresponding server based on the correspondence between each data change event and the server. Finally, save each data change event, its offset, and the server identifier labeled for each data change event to a data table.

[0091] In practical implementation, it is assumed that the database server cluster includes three servers with a master-slave relationship, namely server A, server B and server C. The log file of server A is log a, the log file of server B is log b and the log file of server C is log c, and the contents of the log files of each server are shown in Table 1.

[0092] Table 1

[0093]

[0094] As shown in the table above, the order of the timestamps corresponding to each data change event is timestamp-1 < timestamp-2 < timestamp-3 < timestamp-4 < timestamp-5 < timestamp-6. Therefore, the sorting results of each data change event according to its corresponding timestamp are shown in Table 2.

[0095] Table 2

[0096] Data change events Timestamp Data Change Event 1 Timestamp -1 Data Change Event 3 Timestamp-2 Data Change Event 5 Timestamp -3 Data Change Event 2 Timestamp -4 Data Change Event 4 Timestamp -5 Data Change Event 6 Timestamp - 6

[0097] Then, according to the above arrangement, the displacement offset was set for each data change event, and the results are shown in Table 3.

[0098] Table 3

[0099] offset Data change events Timestamp 1 Data Change Event 1 Timestamp-1 2 Data Change Event 3 Timestamp-2 3 Data Change Event 5 Timestamp -3 4 Data Change Event 2 Timestamp -4 5 Data Change Event 4 Timestamp -5 6 Data Change Event 6 Timestamp - 6

[0100] The correspondence between each data change event and the server is established, and the server identifier to which each data change event belongs is marked. Each data change event, the offset of each data change event, and the server identifier marked for each data change event are saved in the data table shown in Table 4.

[0101] Table 4

[0102]

[0103]

[0104] In practice, since each server provides read and write functionality to users, it's possible for the same data to be modified on different servers. Therefore, before sorting the data change events occurring on multiple servers in ascending order of their timestamps, we can analyze each data change event to determine if there are at least two identical data change events generated by different servers. Here, identical data change events refer to at least two data change events indicating the same data modification operation statements, such as executing the same DML statement on the same data. In other words, the same modification operation was performed on the same data on two servers. If such at least two data change events are confirmed to exist, the data change event with the largest timestamp among the at least two data change events can be retained, and the remaining data change events among the at least two data change events can be deleted. This ensures that the same data change event will not instruct different servers to repeatedly execute the synchronization update process, and retaining the data change event with the largest timestamp also ensures data correctness.

[0105] In step 202, according to the order of the offsets recorded in the data table, perform the following operations for each data change event:

[0106] The data change event is treated as the data change event to be synchronized. Based on the offset of the data change event to be synchronized and the completed offset of each server recorded in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers. The completed offset of any server refers to the offset of the data change event that the server has currently completed synchronizing.

[0107] In specific implementation, taking the database server cluster as an example, which includes three servers with a master-slave relationship, namely server A, server B and server C, and the data table content as shown in Table 4 above, the initial state of the completed offset corresponding to each server recorded in the offset table can be shown in Table 5.

[0108] Table 5

[0109] server offset A 0 B 0 C 0

[0110] As shown in Table 5 above, the offsets of servers A, B, and C recorded in the offset table are all 0, which means that servers A, B, and C have not yet synchronized data for any data change event in Table 4. If server A completes data synchronization for data change event 1, since the offset of data change event 1 in Table 4 is 1, the offset of server A in the offset table will be updated from "0" to "1", and so on.

[0111] In practice, the completed offset for each server recorded in the offset table is as shown in Table 6.

[0112] Table 6

[0113] server offset A 2 B 1 C 1

[0114] As shown in Table 6 above, based on the completed offsets recorded in the offset table for each server, we know that the offset for server A is 2, indicating that server A has completed data synchronization for data change events 1 and 3. The offsets for servers B and C are 1, indicating that servers B and C have completed data synchronization for data change event 1. Therefore, based on the offsets of the data change events to be synchronized, we know that the next data change event to be synchronized for server A is data change event 5, for server B it is data change event 3, and for server C it is data change event 3. Thus, a data synchronization instruction carrying data change event 5 can be sent to server A, and a data synchronization instruction carrying data change event 3 can be sent to servers B and C to synchronize data according to the order of the data change events, thereby achieving data consistency across servers.

[0115] In step 203, a synchronization completion message is received from some or all of the multiple servers, and the completed offset of the corresponding server is updated to the offset of the data change event to be synchronized based on the synchronization completion message.

[0116] In practice, after receiving the data synchronization command from the processing node, each server can first determine whether the data change event to be synchronized carried in the command occurred on its corresponding server. For example, if the data change event to be synchronized carried in the data synchronization command received by server A is data change event 2, since data change event 2 itself occurred by server A, in practical applications, server A does not need to perform data synchronization operation on data change event 2. Therefore, the synchronization completion message can actually be sent by any server when it determines that the data change event to be synchronized has been synchronized, or it can be sent by any server when it determines that the server identifier of the data change event to be synchronized indicated in the data synchronization command is consistent with its own identifier. This can speed up the data synchronization process.

[0117] In practice, since it cannot be guaranteed that synchronization completion messages from some or all of the multiple servers will be received, a preset time can be set to receive synchronization completion messages from some or all of the multiple servers within a preset time. Here, the preset time can be, for example, recording the time required for each server to perform data synchronization when it is running normally, and then taking the maximum value or average value as the preset time. Of course, the preset time can also be determined in other ways, and the specific settings can be made by technical personnel. Then, the timer starts from the time the data synchronization command is sent. If it is determined that no synchronization completion message is received from any server within the preset time, the corresponding processing is performed according to the cause of the server's failure. For example, if the cause of the server's failure is that the server is in a downtime state, the data synchronization command is resent to the server when the server's running status is detected to be normal. If the cause of the server's failure is that the server does not exist, the server and its corresponding offset are deleted from the offset table.

[0118] In practice, the processing node can monitor the operating status of each server in real time and determine the cause of the failure based on the operating status. The operating status includes online status, downtime status, and canceled status. When a server is in the downtime status, it may have received the data synchronization command sent by the processing node but has not yet executed the data synchronization, or it may have completed the data synchronization but has not yet sent the synchronization completion message, or it may have crashed before receiving the data synchronization command. Therefore, the processing node can resend the data synchronization command to the server when it detects that the server's operating status is normal to avoid data inconsistency. When a server is in the canceled status, it means that the server is no longer in use, so there is no need to send the data synchronization command to the server. Therefore, the server and its corresponding offset can be directly deleted from the offset table.

[0119] In practical applications, as the database is used, a large number of data change events will occur on each server. This means that the number of data change events stored in the data table will continue to grow, which will also increase the storage space required by the data table. Therefore, in order to avoid the data change events that have been synchronized occupying the storage space of the data table, the data table can be cleaned up.

[0120] In some embodiments, after saving each data change event and its offset to a data table, when it is determined that the preset cleanup conditions are met, the minimum offset can be determined based on the completed offset recorded in the offset table for each server. Then, based on the minimum offset, the offsets in the data table that are less than or equal to the minimum offset and the corresponding data change events can be cleaned up.

[0121] In practice, pre-set cleanup conditions can be used, such as reaching a preset cleanup cycle or the total number of data change events in the data table reaching a preset number. In this case, the minimum offset can be determined based on the completed offset recorded in the offset table for each server. Then, the offsets in the data table that are less than or equal to the minimum offset and their corresponding data change events are cleaned up. In this way, cleaning up the offsets in the data table that are less than or equal to the minimum offset and their corresponding data change events can ensure that each server does not lose data synchronization for any data change event. This not only cleans up the data table but also further ensures data consistency.

[0122] In this embodiment, multiple servers are in a master-slave relationship, and each server provides read and write functions for users, which can improve data query speed. Moreover, if any database server fails, the other master-slave database servers can still work normally, ensuring data security and achieving high availability of the database. At the same time, a data table and an offset table are set. When a data change event occurs on any database server, the offset of this data change event is set and stored in the data table. Based on the data change events in the data table, each database server is updated synchronously, and the offset of any data change event that has been synchronized on any database server is updated, ensuring data consistency within each database server.

[0123] The data processing method of this application will be described in detail below with reference to specific embodiments.

[0124] Taking a database server cluster consisting of multiple MySQL servers in a master-slave relationship as an example, with the processing node configured as a program capable of reading, writing, adding, deleting, modifying, and querying data on the MySQL servers, and the data tables and offset tables stored in the distributed coordination server ZOOKEEPER, and the log file being a Binlog file, the specific data processing method can be implemented according to the following steps.

[0125] Step 1: Send a request to each MySQL server to retrieve the Binlog file. From the Binlog files sent by each MySQL server, retrieve the data modification events, such as DDL statements and DML statements.

[0126] Step 2: Set an offset for each data change event and store each data change event and its corresponding offset in the ZOOKEEPER data table.

[0127] In practice, ZOOKEEPER can also be a cluster of multiple ZOOKEEPERs. The Binlog file also records key information such as the timestamps of data change events. Therefore, the offset can be set for each data change event according to the order of the timestamps.

[0128] Step 3: Read the offset table of ZOOKEEPER. Based on the offset that has been updated for each MySQL server, determine the data change event corresponding to the next offset of the offset that has been updated for each MySQL server in the data table as the next data change event to be synchronized for the corresponding MySQL server.

[0129] In practice, when determining the data change event corresponding to the next offset of the already updated offsets of each MySQL server within the data table, the server where the data change event occurred can be identified. If the server where the data change event occurred is the same as any other server, then that data change event is skipped, and the next data change event of that event is identified as the next data change event to be synchronized for the corresponding MySQL server. This speeds up data synchronization.

[0130] Step 4: Perform the following steps for each MySQL server.

[0131] The data change event to be synchronized is carried in the data synchronization command and sent to the MySQL server so that the MySQL server can execute the data synchronization process for the data change event to be synchronized.

[0132] Step 5: Receive the synchronization completion message sent by the MySQL server, and update the offset of the MySQL server in the offset table to the offset corresponding to the data change event to be synchronized.

[0133] Figure 3 This is a schematic diagram of the framework of a data processing method provided in an embodiment of this application, as shown below. Figure 3 As shown, Zookeeper has a data table and an offset table. The database server cluster consists of two MySQL servers, MySQL-1 and MySQL-2, which are in a master-slave relationship. Step 1 is to store the data change events obtained from the Binlog file in MySQL-1 into the data table. Step 2 is to send the data change events in the data table to MySQL-2 for data synchronization. Step 3 is to wait for the synchronization completion message returned by MySQL-2 when the data synchronization is completed, and update the offset corresponding to the data change event of MySQL-2 in the offset table to achieve data synchronization.

[0134] In this way, the offset table is updated once after each synchronization, ensuring that the same data change event will not be synchronized repeatedly and that no data change event will be lost.

[0135] In practice, to ensure that the data table does not occupy a large amount of storage space in Zookeeper, data change events in the data table can be cleaned up at certain intervals. The minimum offset in the offset table is selected, and data change events in the data table that are less than or equal to the minimum offset are cleaned up to ensure synchronization consistency.

[0136] Based on the same technical concept, this application also provides a data processing device. The principle of the data processing device in solving the problem is similar to that of the above-described data processing method. Therefore, the implementation of the data processing device can refer to the implementation of the data processing method, and the repeated parts will not be described again.

[0137] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. It is applied to a processing node in a distributed storage system. The distributed storage system includes the processing node and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship. Each server provides read and write functions for users and includes: a setting module 401, an execution module 402, and a receiving module 403.

[0138] The setting module 401 is used to set offsets for each data change event that occurs on each server, and save each data change event and its offset to a data table;

[0139] Execution module 402 is used to perform the following operations for each data change event according to the offset order recorded in the data table:

[0140] The data change event is taken as the data change event to be synchronized, and according to the offset of the data change event to be synchronized and the completed offset of each server recorded in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers; the completed offset of any server refers to the offset of the data change event that the server has currently completed synchronizing.

[0141] The receiving module 403 is used to receive synchronization completion messages sent by some or all of the plurality of servers, and update the completed offset of the corresponding server to the offset of the data change event to be synchronized based on the synchronization completion message.

[0142] In some embodiments, it also includes:

[0143] The acquisition module 404 is used to send a log file retrieval request to each server before setting the offset for each data change event that occurs on each server; the log file request is used to instruct the corresponding server to send the log file to the processing node, and the log file is used to record the data change events that occur on the server.

[0144] Based on the log files received from each server, obtain the data change events that occurred on each server.

[0145] In some embodiments, each data change event carries a timestamp indicating when the data change event occurred; the setting module 401 is specifically used for:

[0146] Sort the data change events that occurred on multiple servers in ascending order of their timestamps;

[0147] Set a unique offset for each data change event based on the order in which they are arranged.

[0148] Based on the correspondence between each data change event and the server, the server identifier to which each data change event belongs is marked;

[0149] Save each data change event, the offset of each data change event, and the server identifier labeled for each data change event to the data table.

[0150] In some embodiments, before the setting module 401 sorts the data change events occurring on multiple servers in ascending order of their timestamps, it further includes:

[0151] The judgment module 405 is used to determine whether there are at least two identical data change events that were generated by different servers.

[0152] If it is determined that there exists, the data change event with the largest timestamp among the at least two data change events is retained, and the remaining data change events among the at least two data change events are deleted.

[0153] In some embodiments, the synchronization completion message is sent by any server when it determines that the synchronization of the data change event to be synchronized is complete, or by any server when it determines that the server identifier of the data change event to be synchronized indicated in the data synchronization instruction is consistent with the identifier of that server.

[0154] In some embodiments, after the setting module 401 saves each data change event and the offset of each data change event to the data table, it further includes:

[0155] The cleaning module 406 is used to determine the minimum offset based on the completed offset corresponding to each server recorded in the offset table when it is determined that the preset cleaning conditions are met.

[0156] Based on the minimum offset, offsets in the data table that are less than or equal to the minimum offset and their corresponding data change events are cleaned up.

[0157] In some embodiments, the receiving module 403 is specifically used for:

[0158] Within a preset time period, receive synchronization completion messages sent by some or all of the multiple servers;

[0159] If it is determined that no synchronization completion message is received from any server within the preset time, then the corresponding processing is performed according to the cause of the server failure.

[0160] In some embodiments, the receiving module 403 is specifically used for:

[0161] If the server malfunction is due to the server being down, then once the server is detected to be running normally, the data synchronization command will be resent to the server.

[0162] If the server malfunctions because the server does not exist, then the server and its corresponding offset are deleted from the offset table.

[0163] The module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. Coupling between modules can be achieved through interfaces, typically electrical communication interfaces, but mechanical interfaces or other types of interfaces are also possible. Therefore, modules described as separate components may or may not be physically separate; they can be located in one place or distributed across different locations on the same or different devices. The integrated modules described above can be implemented in hardware or as software functional modules.

[0164] After introducing the data processing method and apparatus according to exemplary embodiments of this application, an electronic device according to another exemplary embodiment of this application will be described next.

[0165] The following reference Figure 5 To describe an electronic device 130 implemented according to this embodiment of the present application. Figure 5The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0166] like Figure 5 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0167] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0168] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0169] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0170] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0171] In an exemplary embodiment, a storage medium is also provided, which enables the electronic device to perform the data processing method described above when a computer program in the storage medium is executed by a processor of the electronic device. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0172] In an exemplary embodiment, the electronic device of this application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, it enables the at least one processor to perform the steps of any data processing method provided in the embodiments of this application.

[0173] In an exemplary embodiment, a computer program product is also provided, which, when executed by an electronic device, enables the electronic device to implement any of the exemplary methods provided in this application.

[0174] Furthermore, computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0175] The program product for data processing in the embodiments of this application may be a CD-ROM and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0176] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0177] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency (RF), or any suitable combination thereof.

[0178] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0179] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0180] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0181] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0185] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0186] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method applied to a processing node in a distributed storage system, characterized in that, The distributed storage system includes the processing nodes and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship, and each server provides read and write functions to users. The method includes: Set an offset for each data change event that occurs on each server, and save each data change event and its offset to a data table; For each data change event, perform the following operations according to the order of the offsets recorded in the data table: The data change event is taken as the data change event to be synchronized, and according to the offset of the data change event to be synchronized and the completed offset of each server recorded in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers; the completed offset of any server refers to the offset of the data change event that the server has currently completed synchronizing. Receive synchronization completion messages from some or all of the multiple servers, and update the completed offset of the corresponding server to the offset of the data change event to be synchronized based on the synchronization completion message; Each data change event carries a timestamp indicating when the data change event occurred; The step of setting offsets for data change events occurring on each server and saving each data change event and its offset to a data table includes: Sort the data change events that occurred on multiple servers in ascending order of their timestamps; Set a unique offset for each data change event based on the order in which they are arranged. Based on the correspondence between each data change event and the server, each data change event is labeled with the server identifier to which it belongs; Save each data change event, the offset of each data change event, and the server identifier labeled for each data change event to the data table; Before sorting the data change events occurring on multiple servers in ascending order of their timestamps, the method further includes: Determine if there are at least two identical data change events that were generated by different servers; If it is determined that such an event exists, the data change event with the largest timestamp among the at least two data change events is retained, and the remaining data change events among the at least two data change events are deleted.

2. The method as described in claim 1, characterized in that, Before setting the offset for each data change event occurring on each server, the method further includes: Each server is sent a request to retrieve a log file; the log file request is used to instruct the corresponding server to send a log file to the processing node, and the log file is used to record data change events that occur on the server. Based on the log files received from each server, obtain the data change events that occurred on each server.

3. The method as described in claim 1, characterized in that, The synchronization completion message is sent by any server when it determines that the synchronization of the data change event to be synchronized is complete, or by any server when it determines that the server identifier of the data change event to be synchronized indicated in the data synchronization instruction is consistent with the identifier of that server.

4. The method as described in claim 1, characterized in that, After saving each data change event and its offset to the data table, the method further includes: When it is determined that the preset cleanup conditions are met, the minimum offset is determined based on the completed offset corresponding to each server recorded in the offset table. Based on the minimum offset, offsets in the data table that are less than or equal to the minimum offset and their corresponding data change events are cleaned up.

5. The method as described in claim 1, characterized in that, The method for receiving synchronization completion messages from some or all of the plurality of servers includes: Within a preset time period, receive synchronization completion messages sent by some or all of the multiple servers; If it is determined that no synchronization completion message is received from any server within the preset time, then the corresponding processing is performed according to the cause of the server failure.

6. The method as described in claim 5, characterized in that, The method of performing corresponding processing based on the cause of the server failure includes: If the server malfunction is due to the server being down, then once the server is detected to be running normally, the data synchronization command will be resent to the server. If the server malfunctions because the server does not exist, then the server and its corresponding offset are deleted from the offset table.

7. A data processing apparatus, applied to a processing node in a distributed storage system, characterized in that, The distributed storage system includes the processing nodes and a database server cluster. The database server cluster includes multiple servers in a master-slave relationship, and each server provides read and write functions to users, including: The settings module is used to set offsets for each data change event that occurs on each server, and save each data change event and its offset to a data table; The execution module is used to perform the following operations for each data change event, according to the offset order recorded in the data table: The data change event is taken as the data change event to be synchronized, and according to the offset of the data change event to be synchronized and the completed offset of each server recorded in the offset table, a data synchronization instruction for the data change event to be synchronized is sent to some or all of the multiple servers; the completed offset of any server refers to the offset of the data change event that the server has currently completed synchronizing. The receiving module is used to receive synchronization completion messages sent by some or all of the multiple servers, and update the completed offset of the corresponding server to the offset of the data change event to be synchronized based on the synchronization completion message. Each data change event carries a timestamp indicating when the data change event occurred; the setting module is specifically used to: sort the data change events occurring on multiple servers in ascending order of their timestamps; set a unique offset for each data change event according to their sorting order; label each data change event with its corresponding server based on the correspondence between each data change event and the server; and save each data change event, its offset, and the server identifier labeled for each data change event to a data table. The judgment module is used to determine whether there are at least two identical data change events that were generated by different servers before sorting the data change events on multiple servers according to the data change event timestamps in ascending order; if it is determined that there are, the data change event with the largest timestamp among the at least two data change events is retained, and the remaining data change events among the at least two data change events are deleted.

8. An electronic device, characterized in that, It includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer, which, when run on the computer, causes the computer to perform the method as described in any one of claims 1 to 6.

Images