Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment, and relates to the technical field of cloud computing, and the method comprises the steps that under the condition that a data definition language DDL operation is executed, a redo log is generated, and the redo log comprises a first operation log for a data file and a metadata file and a modification log for memory information; sending the redo log and the synchronization request to a storage node; and under the condition that a first analysis result sent by the storage node in response to the synchronization request is received, sending the redo log to a slave computing node. According to the embodiment of the invention, the redo log of the metadata file is stored in the storage node, and the slave computing node can obtain the redo log of the DDL operation from the storage node under the condition that the slave computing node and the master computing node are crashed successively, so that the condition of data loss is avoided.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data processing method, device and electronic device. Background Art

[0002] With the development of cloud computing, database migration to the cloud has become a trend. Cloud native databases usually use the master computing node as a read-write node (RW) to process read-write requests, and the slave computing node as a read-only node (RO) to process read-only requests. All storage tasks are pushed down to the storage node (DN). In the above architecture, generally speaking, data files are stored in storage nodes, and metadata files are stored in computing nodes.

[0003] Based on the above architecture, in some high-availability scenarios, if the master computing node crashes abnormally, a slave computing node with the latest log sequence number (LSN) will be found among the slave computing nodes, and its role will be promoted to the master computing node.

[0004] However, in a scenario where a slave compute node crashes abnormally and fails to restart, and then the master compute node also crashes abnormally after successfully executing some Data Definition Language (DDL) operations, according to the above master selection strategy, if the slave compute node that crashed previously is selected as the master compute node to start, the newly promoted master compute node may suffer from data loss because the latest DDL operation executed did not synchronize the metadata file to the slave compute node that crashed abnormally. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, device and electronic device to solve the problem of data loss in a newly promoted master computing node in a scenario where the original slave computing node and the master computing node fail successively.

[0006] To solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a data processing method, which is executed by a main computing node, and the method includes:

[0008] When a data definition language DDL operation is executed, a redo log is generated, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0009] Sending the redo log and synchronization request to the storage node;

[0010] In case of receiving the first parsing result sent by the storage node in response to the synchronization request, the redo log is sent to the slave computing node.

[0011] Optionally, the slave computing nodes are all slave computing nodes in a working state, and the method further includes:

[0012] receiving the second parsing results sent by all the slave computing nodes, wherein the second parsing results include second parsing information of the redo log;

[0013] Performing verification processing on the second parsing results of all the slave computing nodes based on the second parsing information to obtain a verification result;

[0014] An operation on the DDL is determined based on the verification result.

[0015] Optionally, determining an operation on the DDL based on the verification result includes:

[0016] If the second parsing results of all the slave computing nodes are successfully verified, performing operations after the DDL operation;

[0017] In the case where the second parsing result verification of any slave computing node among all the slave computing nodes fails, a prompt message of an error log is output.

[0018] Optionally, the first parsing result includes first parsing information of the first operation log; after sending the redo log and the synchronization request to the storage node, the method further includes:

[0019] Upon receiving the first parsing result sent by the storage node in response to the synchronization request, verifying the first parsing information;

[0020] In the case where the first parsed information of more than half of the storage nodes among all the storage nodes is verified, it is determined that the verification of all the storage nodes is completed.

[0021] Optionally, sending the redo log to the slave computing node includes:

[0022] Storing the redo log in a preset redo log buffer;

[0023] Receiving the log request sent from the computing node;

[0024] In response to the log request, the redo log in the redo log buffer is sent to the slave computing node.

[0025] In a second aspect, an embodiment of the present application provides a data processing method, which is executed by a storage node, and the method includes:

[0026] Receive a redo log and a synchronization request sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0027] In response to the synchronization request, the redo log is stored, and the first operation log is parsed to obtain a first parsing result, wherein the first parsing result includes first parsing information of the first operation log;

[0028] The first parsing result is sent to the main computing node.

[0029] In a third aspect, an embodiment of the present application provides a data processing method, which is executed by a slave computing node, and the method includes:

[0030] Receive a redo log sent by the primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0031] Parsing the redo log to obtain a second parsing result, wherein the second parsing result includes parsing information of the redo log;

[0032] Send the second parsing result to the main computing node.

[0033] Optionally, before receiving the redo log sent by the primary computing node, the method further includes:

[0034] Send a log request to the primary computing node.

[0035] In a fourth aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0036] A generating module, used for generating a redo log when executing a data definition language DDL operation, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0037] A first sending module, used for sending the redo log and synchronization request to a storage node;

[0038] The second sending module is used to send the redo log to the slave computing node when receiving the first parsing result sent by the storage node in response to the synchronization request.

[0039] In a fifth aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0040] A receiving module, configured to receive a redo log and a synchronization request sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0041] a storage module, configured to store the redo log in response to the synchronization request, and parse the first operation log to obtain a first parsing result, wherein the first parsing result includes first parsing information of the first operation log;

[0042] A sending module is used to send the first parsing result to the main computing node.

[0043] In a sixth aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0044] A receiving module, configured to receive a redo log sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0045] A parsing module, configured to parse the redo log to obtain a second parsing result, wherein the second parsing result includes parsing information of the redo log;

[0046] A sending module is used to send the second parsing result to the main computing node.

[0047] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data processing method described in the first to third aspects.

[0048] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described in the first to third aspects are implemented.

[0049] In a ninth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the data processing method as described in the first to third aspects.

[0050] In an embodiment of the present application, the redo log of the metadata file is stored in the storage node. When the slave computing node and the master computing node fail successively, the slave computing node can obtain the redo log of the DDL operation from the storage node without causing data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 It is one of the flow charts of a data processing method provided in an embodiment of the present application;

[0053] Figure 2 This is a schematic diagram of the architecture of a cloud native database provided in an embodiment of the present application;

[0054] Figure 3 This is one of the data storage schematic diagrams provided in the embodiments of the present application;

[0055] Figure 4 This is the second flowchart of a data processing method provided in an embodiment of the present application;

[0056] Figure 5 This is the third flowchart of a data processing method provided in an embodiment of the present application;

[0057] Figure 6 This is a second data storage schematic diagram provided in an embodiment of the present application;

[0058] Figure 7 This is a fourth flowchart of a data processing method provided in an embodiment of the present application;

[0059] Figure 8 is a schematic diagram of a data loss scenario before implementing the method of the embodiment of the present application;

[0060] Fig. 9 is a schematic diagram of a scenario in which data is not lost after implementing the method of the embodiment of the present application;

[0061] Fig.10 It is one of the structural schematic diagrams of a data processing device provided in an embodiment of the present application;

[0062] Fig.11 This is a second structural diagram of a data processing device provided in an embodiment of the present application;

[0063] Fig.12 This is a third structural diagram of a data processing device provided in an embodiment of the present application;

[0064] Fig.13 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] The embodiments of the present application provide a data processing method, device and electronic device to solve the problem of data loss in a newly promoted master computing node in a scenario where the original slave computing node and the master computing node fail successively.

[0067] See also Figure 1 , Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application, which is executed by the main computing node. Figure 1 As shown, the method comprises the following steps:

[0068] Step 101: When executing a data definition language DDL operation, a redo log is generated, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0069] Step 102: Send the redo log and synchronization request to the storage node;

[0070] Step 103: upon receiving the first parsing result sent by the storage node in response to the synchronization request, send the redo log to the slave computing node.

[0071] The master computing node generates redo logs during the execution of DDL operations. The redo logs include the creation, renaming, and deletion of data files and metadata files, as well as the modification of memory information.

[0072] Among them, metadata files and redo logs related to memory information are newly added log types. The data segment in the redo log of the metadata file stores information such as the file name, file name length, open mode flag, metadata, etc. The redo log of memory information includes modifications to the dict table and TABLE CACHE memory structures.

[0073] The main computing node sends the redo log to the storage node and sends a synchronization request, so that the storage node parses the redo log according to the redo log and the synchronization request, and synchronizes the data of the main computing node. The storage node performs log parsing, including file modification, without modifying the memory information. The storage node sends the first parsing result to the main computing node, and the first parsing result includes the request number, redo log type, tablespace number, parsing success flag, etc.

[0074] The synchronization request is initiated through the network communication module of the main computing node. The main computing node will wait for the storage node to parse the log before returning a response message to the main computing node.

[0075] To further prevent data loss, redo logs and synchronization requests can be sent to multiple storage nodes so that data is stored in multiple storage nodes at the same time. When a storage node fails, data loss will not occur.

[0076] In some optional implementations, when it is determined that the storage node has completed parsing the redo log, the DDL operation data and redo log of the master computing node are stored in the storage node, and the master computing node sends the redo log to the slave computing node.

[0077] In some optional implementations, when the master computing node determines that the storage node has successfully or failed to parse the redo log, the master computing node stores the redo log in a buffer and sends the redo log to the slave computing node; or after the master computing node determines that the storage node has successfully parsed the redo log, the master computing node further verifies the parsed information, and when the parsed information is verified, the master computing node stores the redo log in a buffer and sends the redo log to the slave computing node.

[0078] In some optional implementations, the master computing node sends redo logs to all slave computing nodes, or the master computing node sends redo logs to all slave computing nodes in working state, or the master computing node sends redo logs to a target number of slave computing nodes, where the target number may be a number specified by the master computing node or a preset number, etc.

[0079] In some optional implementations, after receiving the log request sent by the slave computing node, the master computing node sends the redo log to the slave computing node that sent the log request; or the master computing node actively sends the redo log to the slave computing node.

[0080] The architecture of cloud native database is as follows Figure 2 As shown in Figure 1, the master computing node processes read and write requests, the slave computing node processes read-only requests, and all storage tasks are in the storage node. Figure 3 As shown, the data files (*.ibd) are stored in the storage nodes due to the storage separation architecture, while the metadata files (*.frm) are stored in the computing nodes.

[0081] Based on the above architecture and storage method, when the slave computing nodes fail successively, there will be data inconsistency between the master computing node and the slave computing node.

[0082] In the embodiment of the present application, a new log type is added to record DDL operations in the redo log, which conforms to the concept that logs are databases. The metadata file is stored in the storage node. If the computing node crashes, the metadata will not be lost after restarting, and the data consistency will not be affected. It can still be applied in high-availability scenarios such as after the computing node hangs up.

[0083] In the embodiment of the present application, the modification operation of the metadata file is reproduced in the storage node, and the data file and the metadata file are only stored in the storage node, so as to achieve true storage and computing separation. In the case that the slave computing node and the master computing node fail successively, the master computing node and the slave computing node can obtain the DDL operation data and redo log from the storage node, and no data loss will occur.

[0084] Optionally, in some embodiments, the slave computing nodes are all slave computing nodes in a working state, and the method further includes:

[0085] receiving the second parsing results sent by all the slave computing nodes, wherein the second parsing results include second parsing information of the redo log;

[0086] Performing verification processing on the second parsing results of all the slave computing nodes based on the second parsing information to obtain a verification result;

[0087] An operation on the DDL is determined based on the verification result.

[0088] Among them, the slave computing node in working state can be understood as a surviving slave computing node. After each slave computing node in working state receives the redo log sent by the master computing node, it parses the redo log to obtain the second parsing information. The slave computing node performs log parsing, including file modification and memory modification, wherein file modification includes creation, renaming, and deletion of data files and metadata files, and memory modification includes modification of dict table and TABLECACHE memory structure.

[0089] The master computing node receives feedback information including the second parsing information sent by at least one slave computing node, and verifies each slave computing node, that is, compares whether the information saved when the master computing node sends the redo log is completely consistent with the second parsing information fed back by the slave computing node. The second parsing result information of the slave computing node is the same as the first parsing result information of the storage node, including the request number, the redo log type, the table space number, the parsing success flag, etc.

[0090] Optionally, in some embodiments, determining an operation on the DDL based on the verification result includes:

[0091] If the second parsing results of all the slave computing nodes are successfully verified, performing operations after the DDL operation;

[0092] In the case where the second parsing result verification of any slave computing node among all the slave computing nodes fails, a prompt message of an error log is output.

[0093] When the second parsing results of all the slave computing nodes in working state are verified successfully, the logic after the DDL operation is executed.

[0094] When the second parsing result verification of any slave computing node in working state fails, an error log is reported and the error is returned to the upper layer.

[0095] Through the above method, it is possible to ensure that all slave computing nodes synchronize the data of the master computing node, maintain data consistency, and prevent data loss.

[0096] Optionally, in some embodiments, the first parsing result includes first parsing information of the first operation log; after sending the redo log and the synchronization request to the storage node, the method further includes:

[0097] Upon receiving the first parsing result sent by the storage node in response to the synchronization request, verifying the first parsing information;

[0098] In the case where the first parsed information of more than half of the storage nodes among all the storage nodes is verified, it is determined that the verification of all the storage nodes is completed.

[0099] When the master computing node determines that the storage node has completed parsing, the first parsing result is verified, that is, the verification result is determined by comparing whether the information saved when the master computing node sends the redo log to the storage node and the feedback information received from the storage node are completely consistent. If the verification is successful, the verification success count is increased, and if the verification fails, an error is returned to the upper layer. The parsing completion may include the case of parsing success or parsing failure.

[0100] Among them, more than half of the storage nodes can be understood as the majority of storage nodes among all storage nodes, and more than half can be expressed by the formula N / 2+1, where the calculation result of N / 2+1 is rounded down, and N represents the total number of storage nodes.

[0101] In an optional implementation, the main computing node receives feedback information including parsing results from more than half of the storage nodes, verifies and determines that the storage nodes have parsed the log, and then the main computing node saves the redo log to its own redo log buffer, waiting to obtain the redo log from the computing node and parse it.

[0102] In an optional implementation, after the main computing node completes verification of parsed information of more than half of the storage nodes, it is determined that verification of all storage nodes is completed, and subsequent operations can be further performed.

[0103] In an optional implementation, after determining that the storage node has parsed the log and verified it successfully, the master computing node sends the redo log to the slave computing node; if the verification fails, an error is returned to the upper layer.

[0104] By verifying the data, the accuracy of data storage in the storage node can be improved, thereby ensuring the data consistency between the storage node and the main computing node.

[0105] For the redo logs of storage nodes, a persistence strategy is adopted, that is, the majority (i.e. more than half) of the storage nodes write and log synchronization technology is used to ensure that the redo logs are written to the majority of storage nodes before they can be successfully returned. If there is a log loss between storage nodes, the log synchronization technology is used to complete the log so that all storage nodes store and synchronize the redo logs. The majority write strategy ensures that the logs are written to the disk, and the log completion technology ensures the integrity of the logs and increases the reliability of the data.

[0106] For all storage nodes, if there is a read request during DDL, it can be returned as long as it is resolved successfully in any storage node. It can tolerate the situation where some storage nodes are lagging behind. If there is a read request during DDL, it can be returned as long as any storage node satisfies it, making the query more efficient and faster.

[0107] Optionally, in some embodiments, sending the redo log to the slave computing node includes:

[0108] Storing the redo log in a preset redo log buffer;

[0109] Receiving the log request sent from the computing node;

[0110] In response to the log request, the redo log in the redo log buffer is sent to the slave computing node.

[0111] The redo log buffer is a separately applied memory buffer used to store logs generated by the master computing node over a period of time for acquisition by the slave computing node.

[0112] In an optional implementation, the slave computing node may periodically send a log request to the master computing node. The slave computing node may be all computing nodes in a working state.

[0113] See also Figure 4 , Figure 4is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method is executed by a storage node and includes:

[0114] Step 401: Receive a redo log and a synchronization request sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0115] Step 402: In response to the synchronization request, store the redo log, and parse the first operation log to obtain a first parsing result, wherein the first parsing result includes first parsing information of the first operation log;

[0116] Step 403: Send the first parsing result to the main computing node.

[0117] For the redo log and synchronization request, please refer to the description in the above embodiment.

[0118] The storage node stores the redo log and parses the first operation log, and sends a feedback message including the first parsing result to the main computing node, so that the main computing node performs verification according to the first parsing information, thereby improving the accuracy of the data stored in the storage node and the consistency with the data of the main computing node.

[0119] See also Figure 5 , Figure 5 is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method is executed by a slave computing node and includes:

[0120] Step 501: Receive a redo log sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0121] Step 502: parse the redo log to obtain a second parsing result, where the second parsing result includes parsing information of the redo log;

[0122] Step 503: Send the second parsing result to the main computing node.

[0123] The master computing node sends redo logs to the slave computing node, or the slave computing node actively obtains redo logs from the master computing node. After receiving the redo logs, the slave computing node parses the logs and sends the parsing result information of the slave computing node to the master computing node. The master computing node receives feedback information from all slave computing nodes and verifies each slave computing node. Only after all the currently surviving slave computing nodes have been successfully verified will the logic behind the DDL be executed. As long as any slave computing node fails to verify, the entire DDL process will fail to execute, and an error log will be reported and the error will be returned to the upper layer.

[0124] Optionally, in some implementation modes, before receiving the redo log sent by the primary computing node, the method further includes:

[0125] Send a log request to the primary computing node.

[0126] The slave computing node periodically obtains the redo log from the redo log buffer of the master computing node, which can improve the efficiency of data acquisition and improve the data consistency between the slave computing node and the master computing node.

[0127] The data storage method of the embodiment of the present application is as follows Figure 6 As shown in Figure 2, the interaction process between nodes is as follows: Figure 7 As shown, the following steps are included:

[0128] Step 1: The primary computing node generates a redo log for the DDL operation;

[0129] Step 2: The main computing node persists the redo log to the storage node;

[0130] Step 3: The main computing node sends a synchronization request to the storage node;

[0131] Step 4: The storage node performs log parsing based on the redo log and synchronization request;

[0132] Step 5: The storage node feeds back the log parsing results to the main computing node;

[0133] Step 6: The main computing node performs verification according to the analysis results;

[0134] Step 7: When the parsing verification is successful, the main computing node saves the redo log to the log buffer;

[0135] Step 8: Periodically send request logs from the computing node to the main computing node;

[0136] Step 9: The master computing node sends the latest redo log in the log buffer to the slave computing node;

[0137] Step 10: Perform log parsing on the redo log from the computing node;

[0138] Step 11: Feedback the analysis result information from the computing node to the main computing node;

[0139] Step 12: The main computing node performs verification based on the parsing result information. If the verification fails, an error log is reported and the error is returned to the upper layer; if the verification succeeds, the logic behind the DDL continues to be executed.

[0140] Based on the data processing method before the implementation of this solution, such as Figure 8As shown, when the master computing node creates table t1, the data file (*.ibd) corresponding to table t1 is stored in the storage node, and the metadata file (*.frm) is stored in the master computing node and slave computing nodes 1 and 2.

[0141] At time T1, slave computing node 1 crashes, and then the master computing node crashes after creating table t2. The data file (*.ibd) corresponding to table t2 is saved in the storage node, and the metadata file (*.frm) is saved in the master computing node and slave computing node 2 in normal operation. However, slave computing node 1 does not store the relevant data of table t2 due to the crash.

[0142] At time T2, computing node 1 is promoted to the main computing node. However, the computing node does not store the relevant data files of table t2, resulting in data loss.

[0143] After adopting the data processing method of the embodiment of the present application, Fig. 9 As shown, in the same scenario, when the slave computing node 1 goes down at time T1, and the master computing node also goes down after creating table t2, the metadata file and the data file are both stored in the storage node. The master computing node and the slave computing node 1 can obtain the metadata file from the storage node, which can improve data consistency and prevent data loss.

[0144] Since the original system has data loss and data inconsistency problems in high-availability scenarios, it is impossible to derive a general algorithm for calculating the time consumption of DDL operations. However, the implementation method of the present application, based on the strong synchronization of DDL data with storage and computing separation, can calculate the predicted value T of the actual time consumption of DDL operations on the system through the following algorithm:

[0145] T=2nDL k β

[0146] Among them, n represents the number of computing nodes, L represents the amount of newly added redo logs, k represents the number of storage nodes, D represents the network delay, and β represents the specification weight coefficient of each node. This formula is also applicable to high-availability scenarios, which can verify the feasibility of implementing this solution from the side.

[0147] Although the newly added metadata file logs will cause performance degradation as the number of storage nodes increases, this impact can be ignored because the amount of metadata file data is much smaller than that of database files. At the same time, it can be seen that this system is more sensitive to network latency, so it is recommended to use high-performance network cards as hardware for network communication between computing nodes, computing nodes and storage nodes, and storage nodes, and the weight coefficients between various specifications will be slightly different. In actual production activities, DDL operations are usually performed during business off-peak periods or service outages, and systems based on a method for implementing strong synchronization of DDL data based on storage and computing separation are also applicable.

[0148] Most cloud-native databases with storage-computing separation architectures may encounter the situation where the slave computing nodes and the master computing nodes fail one after another. The embodiments of the present application can help avoid and solve the problems of such products in high-availability scenarios and reduce the failure rate of subsequent product services. Usually, a relatively mature product will perform DDL operations every week. If the failure rate is 2%, there will be about one failure per year. If the number of users is about 100,000, it will cause all users to be unable to use it normally.

[0149] In the embodiment of the present application, the main computing node records the DDL operation process by writing a redo log. After writing the redo log, it is first persisted to all storage nodes, and then a synchronization message request is sent to all storage nodes, waiting for most storage nodes to parse this redo log. After receiving the redo log, the storage node parses the log and feeds back the storage node parsing result information to the main computing node.

[0150] The main computing node receives feedback information from most storage nodes, verifies and confirms that the storage node has parsed this log, and then the main computing node saves the redo log to its own redo log buffer, waiting to obtain the redo log from the computing node and parse it.

[0151] The slave computing node actively obtains the redo log from the master computing node. After receiving the redo log, the slave computing node parses the log and sends the parsing result information of the slave computing node to the master computing node. The master computing node receives the feedback information of all slave computing nodes and verifies each slave computing node. Only after all the slave computing nodes are verified successfully will the logic behind the DDL continue to be executed. As long as any slave computing node fails to verify, the entire DDL process will fail to execute and an error will be returned to the upper layer.

[0152] Through the above method, the problem of data loss of the master and slave computing nodes due to the successive failure of the slave computing node in the above scenario can be avoided, and the reliability of data storage can be improved.

[0153] See also Fig.10 , Fig.10 is a structural diagram of a data processing device provided in an embodiment of the present application, which is applied to a main computing node, such as Fig.10 As shown, the data processing device 1000 includes:

[0154] A generating module 1001 is used to generate a redo log when executing a data definition language DDL operation, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information;

[0155] A first sending module 1002, configured to send the redo log and synchronization request to a storage node;

[0156] The second sending module 1003 is used to send the redo log to the slave computing node when receiving the first parsing result sent by the storage node in response to the synchronization request.

[0157] Optionally, the slave computing nodes are all slave computing nodes in a working state, and the device further includes:

[0158] A receiving module, configured to receive the second parsing results sent by all the slave computing nodes, wherein the second parsing results include second parsing information of the redo log;

[0159] A processing module, configured to perform verification processing on the second parsing results of all the slave computing nodes based on the second parsing information to obtain a verification result;

[0160] The first determination module is used to determine an operation on the DDL based on the verification result.

[0161] Optionally, the determining module includes:

[0162] An execution submodule, configured to execute operations subsequent to the DDL operation if the second parsing results of all the slave computing nodes are successfully verified;

[0163] The output submodule is used to output the prompt information of the error log when the second parsing result verification of any slave computing node among all the slave computing nodes fails.

[0164] Optionally, the first parsing result includes first parsing information of the first operation log; and the device further includes:

[0165] a verification module, configured to verify the first parsing information when receiving the first parsing result sent by the storage node in response to the synchronization request;

[0166] The second determination module is used to determine that the verification of all storage nodes is completed when the verification of the first parsed information of more than half of the storage nodes among all the storage nodes is completed.

[0167] Optionally, the second sending module includes:

[0168] A storage submodule, used for storing the redo log in a preset redo log buffer;

[0169] A receiving submodule, used for receiving the log request sent from the computing node;

[0170] A sending submodule is used to send the redo log in the redo log buffer to the slave computing node in response to the log request.

[0171] The data processing device can realize Figure 1 The various processes implemented in the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0172] See also Fig.11 , Fig.11 is a structural diagram of a data processing device provided in an embodiment of the present application, which is applied to a storage node, such as Fig.11 As shown, the data processing device 1100 includes:

[0173] The receiving module 1101 is used to receive the redo log and synchronization request sent by the primary computing node, wherein the redo log includes the first operation log of the data file and the metadata file, and the modification log of the memory information;

[0174] The storage module 1102 is used to store the redo log in response to the synchronization request, and parse the first operation log to obtain a first parsing result, wherein the first parsing result includes first parsing information of the first operation log;

[0175] The sending module 1103 is used to send the first parsing result to the main computing node.

[0176] The data processing device can realize Figure 4 The various processes implemented in the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0177] See also Fig.12 , Fig.12 is a structural diagram of a data processing device provided in an embodiment of the present application, which is applied to a slave computing node, such as Fig.12 As shown, the data processing device 1200 includes:

[0178] The receiving module 1201 is used to receive the redo log sent by the primary computing node, wherein the redo log includes a first operation log for the data file and the metadata file, and a modification log for the memory information;

[0179] The parsing module 1202 is used to parse the redo log to obtain a second parsing result, wherein the second parsing result includes parsing information of the redo log;

[0180] The sending module 1203 is used to send the second parsing result to the main computing node.

[0181] The data processing device can realize Figure 5 The various processes implemented in the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0182] The data processing device in the embodiment of the present application may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, etc., and may also be a server, a network attached storage (NAS), a personal computer (PC), etc., which is not specifically limited in the embodiment of the present application.

[0183] like Fig.13 As shown, an embodiment of the present application further provides an electronic device 1300, comprising: a processor 1301, a memory 1302, and a program stored in the memory 1302 and executable on the processor 1301. When the program is executed by the processor 1301, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0184] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0185] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0188] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, characterized in that: Executed by the main computing node, including: When a data definition language DDL operation is executed, a redo log is generated, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information; Sending the redo log and synchronization request to the storage node; In case of receiving the first parsing result sent by the storage node in response to the synchronization request, the redo log is sent to the slave computing node.

2. The method according to claim 1, characterized in that The slave computing nodes are all slave computing nodes in a working state, and the method further includes: receiving the second parsing results sent by all the slave computing nodes, wherein the second parsing results include second parsing information of the redo log; Performing verification processing on the second parsing results of all the slave computing nodes based on the second parsing information to obtain a verification result; An operation on the DDL is determined based on the verification result.

3. The method according to claim 2, characterized in that The determining the operation of the DDL based on the verification result includes: If the second parsing results of all the slave computing nodes are successfully verified, performing operations after the DDL operation; In the case where the second parsing result verification of any slave computing node among all the slave computing nodes fails, a prompt message of an error log is output.

4. The method according to any one of claims 1 to 3, characterized in that The first parsing result includes first parsing information of the first operation log; After sending the redo log and the synchronization request to the storage node, the method further includes: Upon receiving the first parsing result sent by the storage node in response to the synchronization request, verifying the first parsing information; In the case where the first parsed information of more than half of the storage nodes among all the storage nodes is verified, it is determined that the verification of all the storage nodes is completed.

5. The method according to any one of claims 1 to 3, characterized in that The sending the redo log to the slave computing node includes: Storing the redo log in a preset redo log buffer; Receiving the log request sent from the computing node; In response to the log request, the redo log in the redo log buffer is sent to the slave computing node.

6. A data processing method, characterized in that: Executed by the storage node, including: Receive a redo log and a synchronization request sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information; In response to the synchronization request, the redo log is stored, and the first operation log is parsed to obtain a first parsing result, wherein the first parsing result includes first parsing information of the first operation log; The first parsing result is sent to the main computing node.

7. A data processing method, characterized in that: Executed by slave compute nodes, including: Receive a redo log sent by the primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information; Parsing the redo log to obtain a second parsing result, wherein the second parsing result includes parsing information of the redo log; Send the second parsing result to the main computing node.

8. The method according to claim 7, characterized in that Before receiving the redo log sent by the primary computing node, the method further includes: Send a log request to the primary computing node.

9. A data processing device, characterized in that: include: A generating module, used for generating a redo log when executing a data definition language DDL operation, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information; A first sending module, used for sending the redo log and synchronization request to a storage node; The second sending module is used to send the redo log to the slave computing node when receiving the first parsing result sent by the storage node in response to the synchronization request.

10. A data processing device, characterized in that: include: A receiving module, configured to receive a redo log and a synchronization request sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information; a storage module, configured to store the redo log in response to the synchronization request, and parse the first operation log to obtain a first parsing result, wherein the first parsing result includes first parsing information of the first operation log; A sending module is used to send the first parsing result to the main computing node.

11. A data processing device, characterized in that: include: A receiving module, configured to receive a redo log sent by a primary computing node, wherein the redo log includes a first operation log for a data file and a metadata file, and a modification log for memory information; A parsing module, configured to parse the redo log to obtain a second parsing result, wherein the second parsing result includes parsing information of the redo log; A sending module is used to send the second parsing result to the main computing node.

12. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the program implements the steps of the data processing method according to any one of claims 1 to 5, or implements the steps of the data processing method according to claim 6, or implements the steps of the data processing method according to claim 7 or 8.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 5, or implements the steps of the data processing method as described in claim 6, or implements the steps of the data processing method as described in claim 7 or 8.

14. A computer program product, characterized in that The method comprises computer instructions which, when executed by a processor, implement the steps of the data processing method as claimed in any one of claims 1 to 5, or implement the steps of the data processing method as claimed in claim 6, or implement the steps of the data processing method as claimed in claim 7 or 8.

Images