Data processing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of distributed databases, and discloses a data processing method and device, equipment and a storage medium. Log information of multiple data nodes is analyzed to obtain multiple pieces of data change information; on the basis of the transaction identification information, grouping, merging and sorting change data in the multiple pieces of data change information to obtain transaction change data corresponding to the multiple transactions; determining data return time based on the log site and the minimum preparation time of the log information; and returning the transaction change data of which the submission time is before the data return time in the plurality of transaction change data. Therefore, multiple pieces of data change information are grouped, merged and sorted, so that the change data can be uniformly output according to transactions and can be output according to a sequence when being output, the transaction change data of which the submission time is before the data return time is returned, and the output change data is ensured to be complete data when the transactions are completed; therefore, global order preserving and atomicity are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed database, and particularly relates to a data processing method and device, equipment and a storage medium. BACKGROUND

[0002] In order to solve the problem of insufficient processing capacity of single-machine databases in horizontal expansion, high concurrency and high capacity scenarios, a distributed database is currently used to distribute data on multiple nodes to improve processing capacity through multiple nodes. However, in a distributed database, a single transaction may involve multiple nodes, and when data changes are obtained from each node and output, it is often impossible to guarantee that the output order of the changed data is consistent with the transaction execution order, and it is also impossible to guarantee that all changed data of a transaction can be uniformly output, that is, the data changes cannot meet the global order preservation and atomicity. SUMMARY

[0003] Therefore, the present application provides a data processing method, device, equipment and storage medium to solve the problem that data changes cannot meet the global order preservation and atomicity.

[0004] In a first aspect, the present application provides a data processing method applied to a computing node of a distributed database, which comprises:

[0005] Obtaining log information of multiple data nodes and respectively analyzing the log information of the multiple data nodes to obtain multiple data change information, wherein the data change information comprises transaction identification information, minimum preparation time, submission time and changed data;

[0006] Grouping and merging and sorting the changed data in the multiple data change information based on the transaction identification information to obtain transaction change data corresponding to multiple transactions;

[0007] Determining a data return time based on a log site of the log information and the minimum preparation time;

[0008] Returning transaction change data in the multiple transaction change data whose submission time is located before the data return time.

[0009] In an optional implementation, the grouping, merging and sorting of the changed data in the multiple data change information based on the transaction identification information to obtain transaction change data corresponding to multiple transactions comprises:

[0010] Grouping the multiple data change information based on the transaction identification information to obtain multiple transaction data groups;

[0011] Respectively merging the changed data in the multiple transaction data groups to obtain multiple transaction change data;

[0012] Sort multiple transaction change data based on the commit time corresponding to the transaction change data.

[0013] In an optional implementation, based on the transaction identification information, the changed data in the multiple data change information are grouped, merged, and sorted to obtain the transaction change data corresponding to the multiple transactions, further comprising:

[0014] If the commit time corresponding to the changed data in the transaction change data is consistent, the commit time corresponding to the changed data is used as the commit time corresponding to the transaction change data;

[0015] If the commit times corresponding to the change data in the transaction change data are inconsistent, the maximum commit time among the commit times corresponding to the change data in the transaction change data is used as the commit time corresponding to the transaction change data.

[0016] In an optional implementation, determining the data return time based on the log location and minimum preparation time of the log information includes:

[0017] Determine the target log information with the largest log location in each of the multiple data nodes;

[0018] The minimum value among the minimum preparation times of the plurality of target log information is determined as the data return time.

[0019] In a second aspect, the present invention provides a data processing method, applied to a data node of a distributed database, the method comprising:

[0020] When a transaction is in the prepare phase, obtain and store the prepare time of the data change;

[0021] Sort the stored preparation times and get the minimum preparation time;

[0022] When the transaction is in the commit phase, the minimum preparation time, transaction identification information, and commit time are obtained and written into the log information. The log information is used by the computing node to determine and return the transaction change data.

[0023] In an optional embodiment, the method further includes:

[0024] After the computing node returns the transaction change data, the preparation time corresponding to the transaction change data is deleted.

[0025] In a third aspect, the present invention provides a data processing device, applied to a computing node of a distributed database, comprising:

[0026] The log information analysis module is configured to obtain log information of the plurality of data nodes, and analyze the log information of the plurality of data nodes respectively to obtain a plurality of data change information, wherein the data change information comprises transaction identification information, minimum preparation time, submission time and changed data;

[0027] The changed data merging module is configured to group and merge the changed data in the plurality of data change information based on the transaction identification information to obtain transaction change data corresponding to the plurality of transactions;

[0028] The return time determination module is configured to determine the data return time based on the log site of the log information and the minimum preparation time.

[0029] The changed data return module is configured to return the transaction change data whose submission time is located before the data return time in the plurality of transaction change data.

[0030] In a fourth aspect, the present application provides a data processing device applied to a data node of a distributed database, which comprises:

[0031] The preparation time storage module is configured to obtain and store the preparation time of the data change when the transaction is in the preparation phase.

[0032] The preparation time sorting module is configured to sort the stored preparation time to obtain the minimum preparation time.

[0033] The log information writing module is configured to obtain the minimum preparation time, the transaction identification information and the submission time when the transaction is in the submission phase, and write the minimum preparation time, the transaction identification information and the submission time into the log information, wherein the log information is used for the computing node to determine and return the transaction change data.

[0034] In a fifth aspect, the present application provides a computer device, which comprises a memory and a processor, the memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data processing method of the first aspect or the second aspect or any corresponding embodiment thereof.

[0035] In a sixth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used for making a computer execute the data processing method of the first aspect or the second aspect or any corresponding embodiment thereof.

[0036] In a seventh aspect, the present application provides a computer program product, which comprises computer instructions, and the computer instructions are used for making a computer execute the data processing method of the first aspect or the second aspect or any corresponding embodiment thereof.

[0037] The data processing method provided by the embodiment of the present application groups and combines and sorts the changed data in the plurality of data change information based on transaction identification information, so that the changed data can be output uniformly according to the transaction and can be output according to the sequence; and the data return time is determined based on the log site and the minimum preparation time of the log information, and the transaction changed data with the submission time before the data return time is returned, so that the output changed data is the complete data at the time of transaction completion, thereby guaranteeing the global order preservation and atomicity of the data change output. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is a flowchart of a data processing method according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of another data processing method according to an embodiment of the present application;

[0041] Figure 3 is a flowchart of a data processing method according to an embodiment of the present application;

[0042] Figure 4 is a structural block diagram of a data processing device according to an embodiment of the present application;

[0043] Figure 5 is a structural block diagram of a data processing device according to an embodiment of the present application;

[0044] Figure 6 is a hardware structure schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0046] In order to solve the problem that a single machine database is insufficient in processing capacity in the scene of horizontal expansion, high concurrency and high capacity, a distributed database is usually used to distribute data on multiple nodes to improve the processing capacity through multiple nodes. However, in the distributed database, a single transaction may involve multiple nodes, and when data changes are obtained from each node and output, the output order of the changed data cannot be guaranteed to be consistent with the execution order of the transaction, and all changed data of a transaction cannot be guaranteed to be output uniformly, that is, the data changes cannot meet the global order preservation and atomicity. The global order preservation refers to that the data output order is consistent with the transaction execution order, and the atomicity refers to that all changed data belonging to the same transaction are merged into the same transaction when output.

[0047] In order to solve the above problem, the embodiment of the present application provides a data processing method, which groups and merges and sorts changed data in multiple data change information based on transaction identification information, so that the changed data can be output uniformly according to the transaction and can be output in order when the changed data is output; and determines the data return time based on the log site and the minimum preparation time of the log information, and returns the transaction changed data whose submission time is located before the data return time, so as to ensure that the output changed data is the complete data at the time when the transaction is completed, thereby ensuring the global order preservation and atomicity of the data change output.

[0048] According to the embodiment of the present application, a data processing method embodiment is provided, and it should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] In one aspect, a data processing method is provided in the embodiment, which can be used in a computing node of a distributed database, Figure 1 The flowchart of the data processing method according to the embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0050] In step S101, log information of multiple data nodes is obtained, and the log information of the multiple data nodes is parsed respectively to obtain multiple data change information.

[0051] In the embodiment of the present application, the data node is used to store data, and the data change occurred in the data node is embodied in the form of a log. The computing node obtains the log information of multiple data nodes, and thus the data change occurred in the multiple data nodes can be obtained.

[0052] ​In the embodiment of the present application, the data change information includes transaction identification information, minimum preparation time, commit time and changed data. The transaction identification information is the gid (group id) of the transaction, which is globally unique and used to uniquely identify a transaction; the minimum preparation time is the minimum value of the time stamp used in the preparation phase for all data changes related to a transaction in a data node, that is, the time of the earliest data change for a transaction in a data node; the commit time is the time stamp used in the commit phase for data change; and the changed data is the incremental data corresponding to the data change.

[0053] It should be noted that in the distributed database, the transaction flow of the distributed transaction is divided into two phases, specifically including a preparation phase and a commit phase. In the preparation phase, each participant of the transaction, that is, the data node, locally executes the transaction and generates log information, at this time the transaction is not committed, and the preparation time is the time stamp recorded when the transaction is executed in this phase; in the commit phase, the commit operation of the transaction is executed, and the commit time is the time stamp recorded when the transaction is committed.

[0054] In step S102, the changed data in the plurality of data change information is grouped, merged and sorted based on the transaction identification information, to obtain transaction change data corresponding to a plurality of transactions.

[0055] In the embodiment of the present application, the changed data in the plurality of data change information is grouped, merged and sorted based on the transaction identification information, to obtain transaction change data corresponding to a plurality of transactions. Specifically, the changed data corresponding to the same transaction identification information is divided into a group, and the changed data in the same group is merged to obtain transaction change data, thereby merging all changes belonging to the same transaction into the same transaction, and ensuring the atomicity of data change.

[0056] In the embodiment of the present application, the grouped and merged transaction change data is sorted, so that the transaction change data can be output in the sorted order when the data is returned, and the global order preservation of data change is ensured.

[0057] In step S103, the data return time is determined based on the log site of the log information and the minimum preparation time.

[0058] In the embodiment of the present application, the log site of the log information refers to LSN (Log Sequence Number), which has global uniqueness in the log and gradually increases with the writing of the log. In the plurality of log information, the log information with the largest log site is the last written log information, corresponding to the last performed data change.

[0059] In the embodiment of the present application, the data return time is used to limit the time limit for returning the transaction change data. If the commit times corresponding to the transaction change data are all located before the data return time, it indicates that all the change data corresponding to the transaction has been completely acquired and merged into the transaction change data, and thus the transaction change data can be returned. If the commit times corresponding to the transaction change data are not all located before the data return time, it indicates that all the change data corresponding to the transaction has not been completely acquired and merged into the transaction change data, and thus the transaction change data cannot be returned, and the other change data corresponding to the transaction needs to be continuously waited to be merged into the transaction change data.

[0060] In an optional embodiment, in order to ensure the reliability of the data return time, the data return time can be determined based on the minimum preparation time of the target log information with the largest log site. Specifically, the target log information with the largest log site in each data node is determined as the target log information of the data node, that is, the last written log information in each data node is determined as the target log information of the data node; and the minimum value in the minimum preparation times of the multiple target log information is determined as the data return time.

[0061] In step S104, the transaction change data with the commit time located before the data return time in the multiple transaction change data is returned.

[0062] In the embodiment of the present application, the transaction change data with the commit time located before the data return time in the multiple transaction change data obtained by merging has already merged all the change data, and thus the transaction change data is returned.

[0063] The data processing method provided in the embodiment of the present application groups and merges the change data in the multiple data change information based on the transaction identification information, and sorts the change data, so that the change data can be uniformly output according to the transaction and can be sequentially output when the change data is output. Meanwhile, the data return time is determined based on the log site and the minimum preparation time of the log information, and the transaction change data with the commit time located before the data return time is returned, so that the output change data is the complete data at the completion of the transaction, thereby ensuring the global order preservation and atomicity of the data change output.

[0064] In the embodiment, a data processing method is provided, which can be used in a computing node of a distributed database, Figure 2 is a flowchart of another data processing method according to the embodiment of the present application, as shown in the figure, the flowchart includes the following steps: Figure 2

[0065] In step S201, the log information of multiple data nodes is acquired, and the log information of the multiple data nodes is parsed respectively to obtain multiple data change information. For details, please refer to Figure 1 ​Step S101 of the illustrated embodiment will not be described in detail here.

[0066] Step S202 : based on the transaction identification information, grouping, merging and sorting the changed data in the multiple data change information to obtain the transaction change data corresponding to the multiple transactions.

[0067] Specifically, step S202 includes:

[0068] Step S2021: Group multiple data change information based on the transaction identification information to obtain multiple transaction data groups.

[0069] In an embodiment of the present invention, data change information corresponding to the same transaction identification information is divided into the same group, and data change information corresponding to different transaction identification information is divided into different groups, so that the data change information is grouped according to the corresponding transaction to obtain transaction data groups corresponding to different transactions.

[0070] Step S2022: Merge the changed data in the multiple transaction data groups to obtain multiple transaction changed data.

[0071] In the embodiment of the present invention, in each transaction data group, the changed data in the data change information contained therein are merged to obtain a plurality of transaction change data.

[0072] Step S2023: sorting the plurality of transaction change data based on the commit time corresponding to the transaction change data.

[0073] In an embodiment of the present invention, multiple transaction change data are sorted based on the commit time corresponding to the transaction change data, so that the transaction change data are arranged in ascending order of commit time, and thus the transaction change data are arranged in the order of transaction execution. When the transaction change data is output and returned, the change data can be output in the order of transaction execution, thereby satisfying global order preservation.

[0074] In an alternative embodiment, if the commit times corresponding to the change data in the transaction change data are consistent, the commit time corresponding to the change data is taken as the commit time corresponding to the transaction change data; if the commit times corresponding to the change data in the transaction change data are inconsistent, the maximum value in the commit times corresponding to the change data in the transaction change data is taken as the commit time corresponding to the transaction change data. Thus, in the case that there are different network delays, clock differences or processing speeds between different data nodes, causing inconsistent commit times of different data nodes, the commit time of the last submitted change data is taken as the commit time of the transaction change data, so as to ensure that all the change data contained in the transaction change data has been submitted at the commit time of the transaction change data, thereby ensuring that all the involved change data can be returned when the transaction change data is returned, and ensuring the atomicity of data change.

[0075] In step S203, the data return time is determined based on the log site of the log information and the minimum preparation time. For details, please refer to Figure 1 In step S103 of the embodiment shown in the figure, no further description is given here.

[0076] In step S204, the transaction change data whose commit time is before the data return time is returned. For details, please refer to Figure 1 In step S104 of the embodiment shown in the figure, no further description is given here.

[0077] The data processing method provided by the embodiment of the present application sorts the transaction change data according to the commit time, so that the change data can be output according to the transaction execution order when the transaction change data is output and returned, thereby meeting the global order preservation.

[0078] On the other hand, in the embodiment, a data processing method is provided, which can be used in the data nodes of a distributed database, Figure 3 is a flowchart of the data processing method according to the embodiment of the present application, which includes the following steps: Figure 3 as shown in the figure, the flowchart includes the following steps:

[0079] In step S301, when the transaction is in the preparation phase, the preparation time of data change is obtained and stored.

[0080] In the embodiment of the present application, when the transaction is in the preparation phase, each data node obtains the time stamp of each data change, obtains the preparation time of each data change, and stores it.

[0081] In step S302, the stored preparation time is sorted to obtain the minimum preparation time.

[0082] In the embodiment of the present application, the one-time preparation time is obtained and stored, the stored preparation times are sorted once, the minimum preparation time is determined from the sorted preparation times, and the minimum preparation time is taken as the minimum preparation time.

[0083] In step S303, when the transaction is in the commit phase, the minimum preparation time, the transaction identification information, and the commit time are obtained, and the minimum preparation time, the transaction identification information, and the commit time are written into the log information.

[0084] In the embodiment of the present application, when the transaction is in the commit phase, each data node obtains the minimum preparation time, the transaction identification information corresponding to the transaction, and the commit time, and writes the minimum preparation time, the transaction identification information corresponding to the transaction, and the commit time into the log information, and the log information is used by the computing node to determine and return the transaction change data.

[0085] In an optional implementation, after the computing node returns the transaction change data, the preparation time corresponding to the transaction change data is deleted.

[0086] In the embodiment, a data processing apparatus is also provided, which is used to implement the above-described embodiments and preferred embodiments, and details are not repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0087] In an aspect, the embodiment provides a data processing apparatus applied to a computing node of a distributed database, as shown in Figure 4 The data processing apparatus comprises:

[0088] The log information parsing module 401 is configured to obtain log information of a plurality of data nodes, and parse the log information of the plurality of data nodes respectively to obtain a plurality of data change information, wherein the data change information comprises transaction identification information, minimum preparation time, commit time, and change data.

[0089] The change data merging module 402 is configured to group and merge and sort the change data in the plurality of data change information based on the transaction identification information to obtain transaction change data corresponding to a plurality of transactions.

[0090] The return time determining module 403 is configured to determine a data return time based on a log site of the log information and the minimum preparation time.

[0091] The change data returning module 404 is configured to return transaction change data in the plurality of transaction change data, whose commit time is located before the data return time.

[0092] In an optional implementation, the change data merging module 402 comprises:

[0093] a data grouping unit, configured to group the multiple pieces of data change information based on the transaction identification information, to obtain multiple pieces of transaction data grouping;

[0094] a data merging unit, configured to respectively merge the change data in the multiple pieces of transaction data grouping, to obtain multiple pieces of transaction change data;

[0095] a data sorting unit, configured to sort the multiple pieces of transaction change data based on the commit time corresponding to the transaction change data.

[0096] In an optional implementation, the change data merging module 402 further includes:

[0097] a commit time determining unit, configured to, if the commit times corresponding to the change data in the transaction change data are consistent, take the commit time corresponding to the change data as the commit time corresponding to the transaction change data;

[0098] the commit time determining unit is further configured to, if the commit times corresponding to the change data in the transaction change data are inconsistent, take the maximum value in the commit times corresponding to the change data in the transaction change data as the commit time corresponding to the transaction change data.

[0099] a target log determining unit, configured to respectively determine target log information corresponding to the maximum log site in the multiple data nodes;

[0100] a return time determining unit, configured to take the minimum value in the minimum preparation times of the multiple pieces of target log information as the data return time.

[0101] On the other hand, the embodiment provides a data processing apparatus applied to a data node of a distributed database, as shown in the following figure, which includes: Figure 5

[0102] a preparation time storage module 501, configured to acquire and store the preparation time of data change when the transaction is in a preparation phase;

[0103] a preparation time sorting module 502, configured to sort the stored preparation time, to obtain a minimum preparation time;

[0104] a log information writing module 503, configured to acquire the minimum preparation time, the transaction identification information and the commit time when the transaction is in a commit phase, and write the minimum preparation time, the transaction identification information and the commit time into log information, the log information being used for a computing node to determine and return transaction change data.

[0105] In an optional implementation, the apparatus further includes:

[0106] ​The preparation time deletion module is used to delete the preparation time corresponding to the transaction change data after the computing node returns the transaction change data.

[0107] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0108] The data processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0109] The embodiment of the present invention also provides a computer device having the above Figure 4 or Figure 5 The data processing device shown.

[0110] See also Figure 6 , Figure 6 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0111] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0112] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0113] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required for at least one function, etc. The data storage area can store data created by the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some alternative embodiments, the memory 20 can optionally include memory that is remotely located with respect to the processor 10, and which can be connected to the computer device through a network. Examples of such networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communications network, and combinations thereof.

[0114] The memory 20 can include a volatile memory, such as a random access memory, and / or can include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid state memory device. The memory 20 can also include an array of multi-state flash memory cells, which can be used to store data and / or instructions in multiple states.

[0115] The computer device also includes a communications interface 30 for communicating with other devices or communication networks.

[0116] The embodiments of the present application also provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or implemented as computer code to be originally stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the method shown in the above embodiments.

[0117] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, can invoke or provide the method and / or technical solutions according to the present application. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0118] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the present application.

Claims

1. A data processing method, characterized in that: Applied to a computing node of a distributed database, the method includes: Obtain log information of multiple data nodes, and parse the log information of the multiple data nodes respectively to obtain multiple data change information, wherein the data change information includes transaction identification information, minimum preparation time, submission time and change data; Based on the transaction identification information, grouping, merging and sorting the changed data in the plurality of data change information to obtain transaction change data corresponding to the plurality of transactions; Determining a data return time based on the log location and minimum preparation time of the log information; Return the transaction change data whose commit time is before the data return time among the plurality of transaction change data.

2. The method according to claim 1, characterized in that The step of grouping, merging, and sorting the changed data in the plurality of data change information based on the transaction identification information to obtain transaction change data corresponding to the plurality of transactions includes: Grouping the plurality of data change information based on the transaction identification information to obtain a plurality of transaction data groups; Merging the changed data in the plurality of transaction data groups respectively to obtain a plurality of transaction changed data; The plurality of transaction change data are sorted based on the commit time corresponding to the transaction change data.

3. The method according to claim 2, characterized in that The step of grouping, merging, and sorting the changed data in the plurality of data change information based on the transaction identification information to obtain transaction change data corresponding to the plurality of transactions further includes: If the commit time corresponding to the change data in the transaction change data is consistent, the commit time corresponding to the change data is used as the commit time corresponding to the transaction change data; If the commit times corresponding to the change data in the transaction change data are inconsistent, the maximum value of the commit times corresponding to the change data in the transaction change data is used as the commit time corresponding to the transaction change data.

4. The method according to claim 1, wherein The determining of the data return time based on the log location and the minimum preparation time of the log information includes: Determine the target log information with the largest corresponding log location among the plurality of data nodes respectively; The minimum value among the minimum preparation times of the plurality of target log information is determined as the data return time.

5. A data processing method, characterized in that: Applied to a data node of a distributed database, the method includes: When a transaction is in the prepare phase, obtain and store the prepare time of the data change; Sort the stored preparation times and get the minimum preparation time; When the transaction is in the commit phase, the minimum preparation time, transaction identification information and commit time are obtained and written into log information, which is used by the computing node to determine and return transaction change data.

6. The method according to claim 5, characterized in that The method further comprises: After the computing node returns the transaction change data, the preparation time corresponding to the transaction change data is deleted.

7. A data processing device, characterized in that: Applied to a computing node of a distributed database, the device comprises: A log information parsing module is used to obtain log information of multiple data nodes and parse the log information of multiple data nodes respectively to obtain multiple data change information, wherein the data change information includes transaction identification information, minimum preparation time, submission time and change data; a change data merging module, configured to group, merge, and sort the change data in the plurality of data change information based on the transaction identification information, to obtain transaction change data corresponding to the plurality of transactions; A return time determination module, configured to determine a data return time based on a log location and a minimum preparation time of the log information; The change data returning module is used to return the transaction change data whose submission time is before the data return time among the multiple transaction change data.

8. A data processing device, characterized in that: Applied to a data node of a distributed database, the device includes: The preparation time storage module is used to obtain and store the preparation time of data changes when the transaction is in the preparation stage; The preparation time sorting module is used to sort the stored preparation times and obtain the minimum preparation time; The log information writing module is used to obtain the minimum preparation time, transaction identification information and submission time when the transaction is in the submission stage, and write the minimum preparation time, transaction identification information and submission time into the log information. The log information is used by the computing node to determine and return the transaction change data.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data processing method according to any one of claims 1 to 4 or 5 to 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method according to any one of claims 1 to 4 or 5 to 6.

Images