A data processing method and device, electronic equipment and computer storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and computer storage medium. Wherein, a mapping relationship queue is determined, the mapping relationship queue contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory; the maximum sequence number submitted by the memory table is determined, and the log number corresponding to the maximum sequence number is queried from the mapping relationship queue; the combination of the log number and the maximum sequence number is stored in the data file corresponding to the memory table; in the case of fault restart, multiple data files contained in the memory are traversed, and the maximum log number is sent to the processing module; the maximum log number is used for the processing module to take the maximum log number as a snapshot point, and data playback is performed according to the snapshot point. In the embodiment, not only the accuracy of the snapshot point can be improved, but also the resource loss of the processing module is reduced.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to a data processing method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] Data availability is crucial across various fields, and for distributed databases, data consistency among the data nodes is particularly important. Currently, various consistency protocols are commonly used to address consistency issues in distributed systems, including two-phase commit, three-phase commit, Paxos, and Raft.

[0003] Taking the Raft protocol as an example, the common approach is to replay the data in each data node by taking a snapshot to achieve data consistency between data nodes. However, this approach usually involves the Raft module performing a snapshot operation based on a set time (such as one day or one week), compressing the logs of a certain data node before that time to generate a snapshot. This snapshot is then used to replay the data of other data nodes based on the snapshot point (the snapshot point is the time point corresponding to the snapshot) when a system failure occurs.

[0004] If a failure occurs after a long period of time since the last snapshot operation, resulting in a large number of new logs on a data node, it would be very resource-intensive and time-consuming to replay the data at the last snapshot point. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and computer storage medium, which can not only improve the accuracy of snapshot points, but also help reduce the resource consumption of Raft modules.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] A mapping relationship queue is determined, which contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory;

[0008] Determine the maximum sequence number submitted by the memory table, and query the log number corresponding to the maximum sequence number from the mapping queue;

[0009] The combination of the log number and the maximum sequence number is stored in the data file corresponding to the memory table;

[0010] In the event of a fault restart, the system traverses multiple data files contained in memory and sends the largest log number to the processing module. The largest log number is used by the processing module as a snapshot point, and data is replayed based on the snapshot point.

[0011] Optionally, the processing module includes a Raft module, and the step of determining the mapping relationship queue includes:

[0012] The data to be submitted and the corresponding log number are determined by the Raft module based on the historical data submitted.

[0013] The data to be submitted is stored in memory, and the sequence number corresponding to the data to be submitted is determined based on the historically stored data.

[0014] Establish a mapping relationship between the log number and the sequence number, and form a mapping relationship queue from multiple mapping relationships.

[0015] Optionally, determining the maximum sequence number submitted by the memory table includes:

[0016] The sequence number corresponding to the most recently stored data in the memory table, arranged in chronological order, is determined to be the maximum sequence number.

[0017] Optionally, storing the combination of the log number and the maximum sequence number into the data file corresponding to the memory table includes:

[0018] If the memory table is persisted to the hard disk, the combination of the log number and the maximum sequence number is stored in the data file corresponding to the memory table.

[0019] Optionally, it also includes:

[0020] If the data contained in the memory table exceeds a preset threshold, the remaining data will be stored in another memory table.

[0021] Secondly, embodiments of this application provide a data processing apparatus, including:

[0022] The first determining module is used to determine the mapping relationship queue, which contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory;

[0023] The second determining module is used to determine the maximum sequence number submitted by the memory table and query the log number corresponding to the maximum sequence number from the mapping relationship queue;

[0024] A storage module is used to store the combination of the log number and the maximum sequence number into the data file corresponding to the memory table;

[0025] The processing module is used to traverse multiple data files contained in memory in the event of a fault restart and send the largest log number to the processing module; the largest log number is used by the processing module as a snapshot point and to perform data replay based on the snapshot point.

[0026] Optionally, the processing module includes a Raft module, and the first determining module is specifically used to determine the data to be submitted and the log number corresponding to the data to be submitted, wherein the log number is determined by the Raft module based on historically submitted data; store the data to be submitted in memory, and determine the sequence number corresponding to the data to be submitted based on historically stored data; establish a mapping relationship between the log number and the sequence number, and form a mapping relationship queue from multiple mapping relationships.

[0027] Optionally, the second determining module is specifically used to determine that the sequence number corresponding to the latest stored data arranged in chronological order in the memory table is the maximum sequence number, and to query the log number corresponding to the maximum sequence number from the mapping relationship queue.

[0028] Optionally, the storage module is specifically used to store the combination of the log number and the maximum sequence number in the data file corresponding to the memory table when the memory table is persisted to the hard disk.

[0029] Optionally, the storage module is further configured to store the remaining acquired data into another memory table if the data contained in the memory table exceeds a preset threshold.

[0030] Thirdly, embodiments of this application provide an electronic device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to implement the data processing method as described in the first aspect above.

[0031] Fourthly, embodiments of this application provide a computer storage medium storing a computer program, which, when executed by a computer, implements the data processing method described in the first aspect above.

[0032] Fifthly, embodiments of this application provide a computer program product that, when run on a computer's processor, causes the computer's processor to execute the data processing method described in the first aspect above.

[0033] In the technical solution provided in this application embodiment, a mapping relationship queue is determined, which contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory; the maximum sequence number submitted by the memory table is determined, and the log number corresponding to the maximum sequence number is retrieved from the mapping relationship queue; the combination of the log number and the maximum sequence number is stored in the data file corresponding to the memory table; in the case of a fault restart, multiple data files contained in the memory are traversed, and the maximum log number is sent to the processing module; the maximum log number is used by the processing module as a snapshot point, and data replay is performed based on the snapshot point, which not only improves the accuracy of the snapshot point, but also helps to reduce the resource consumption of the processing module.

[0034] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0037] Figure 2 A flowchart illustrating another data processing method provided in this application embodiment;

[0038] Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0041] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0042] Before introducing the data processing method provided in the embodiments of this application, a brief introduction will be given regarding the relevant concepts and research background involved in this application:

[0043] (1) Related concepts

[0044] Raft is a consensus protocol for distributed systems that eliminates single points of failure that cause system availability issues. It ensures that the system can still provide services even if less than half of the nodes fail. The Raft distributed consensus protocol uses a state replicator model, replaying the same logs on different data nodes to eventually achieve consistency. The logs contain contiguous data, and replaying the logs involves writing the contiguous data back to the local storage engine in sequence.

[0045] Log Structurally Merged Tree (LSM) storage engine: This is a storage engine that uses a certain strategy to merge incremental data into the full data. The full data refers to all the data contained in the log, while the incremental data refers to the data modified within a certain period of time.

[0046] Idempotency: This means that a user's single or multiple requests for the same operation yield the same result, and there are no side effects from multiple clicks.

[0047] Snapshots are a method used to compress logs. As a system runs longer, the generated logs become increasingly long. When the system crashes or a new node joins, replaying the entire log will consume significant resources and time. The purpose of snapshots is to compress the logs from a specific node, thus greatly reducing the resources and time required for log replay.

[0048] (2) Research Background

[0049] Research has shown that distributed databases store data across multiple data nodes. When data changes on any data node, this change needs to be synchronized to other data nodes to ensure data consistency. Current solutions to the consistency problem in distributed systems typically employ various consistency protocols, including two-phase commit, three-phase commit, Paxos, and Raft. However, data synchronization (i.e., writing the same log to multiple different storage engines) is not effective in addressing resource consumption issues. Therefore, this application proposes a data replay scheme. By determining snapshot points and writing the log data back to the local storage engine sequentially according to these snapshot points, the accuracy of snapshot points is improved, and resource consumption of the Raft module is reduced.

[0050] Specifically, this application focuses on the Raft protocol. Taking Raft as an example, it typically employs a scheme of replaying data from each data node using snapshot points to achieve data consistency between data nodes. However, this scheme usually involves the Raft module performing snapshot operations according to a set time (such as one day or one week), compressing the logs of a data node before that time to generate a snapshot. This snapshot is then used to replay data from other data nodes in the event of a system failure. If a long time has passed since the last snapshot operation when a failure occurs, resulting in a data node accumulating a large number of new logs, replaying the logs from the last snapshot point at this time can easily consume a lot of resources and time.

[0051] To address the aforementioned issues, this application provides a data processing method that involves: determining a mapping queue containing a mapping relationship between log numbers submitted by the processing module and sequence numbers submitted by memory; determining the maximum sequence number submitted by the memory table and retrieving the log number corresponding to the maximum sequence number from the mapping queue; storing the combination of the log number and the maximum sequence number in the data file corresponding to the memory table; in the event of a fault restart, traversing multiple data files contained in memory and sending the maximum log number to the processing module; the maximum log number is used by the processing module as a snapshot point for data replay, which not only improves the accuracy of the snapshot point but also helps reduce the resource consumption of the processing module.

[0052] It should be noted that this application is mainly used in scenarios where data is replayed (i.e., the data contained in the log is written back to the local storage engine in sequence) in the event of a program failure and restart.

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Figure 1 A flowchart of a data processing method provided in an embodiment of this application is shown below. Figure 1 As shown, in this process, from the perspective of the device, the executing entity can be one or more electronic devices; from the perspective of the program, the executing entity can be the program mounted on these electronic devices.

[0055] The method includes:

[0056] 101. Determine the mapping relationship queue, which contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory;

[0057] In this step, the processing module can be the Raft module from an open-source project provided by a third party. It can be integrated into a third-party program. For each data node, the Raft module will generate a corresponding log, and each log will have a corresponding log number, which will increment sequentially.

[0058] Optionally, the memory may include a local disk and store data through an LSM tree storage engine. When the business system receives a data storage service request, it will store the data corresponding to this data storage service request in the memory.

[0059] In this embodiment, the purpose of establishing a mapping queue is that when the Raft protocol needs to perform a snapshot operation at a certain time, all the data currently stored in memory constitutes a snapshot; or when the Raft protocol reaches a certain preset amount of modified data, the modified data constitutes a snapshot. This application considers the data replay scheme via the Raft protocol, typically selecting the most recently established snapshot point (either by compressing the logs of a data node before a certain time, or by compressing the modified data after reaching a preset amount of modified data), and performing data replay based on this snapshot point. If a long time has passed since the last snapshot operation when a fault occurs, resulting in a data node adding a large number of logs, performing data replay based on the last snapshot point would easily consume significant resources and time. Therefore, it is necessary to determine the actual data that needs to be synchronized in memory; this data to be committed can be the logs stored in the Raft module. The method to determine the data to be submitted is to determine the log number corresponding to the data to be submitted. Since the log number is determined by the Raft module and the memory has a certain strategy for calculating the sequence number, it is necessary to establish a mapping relationship between the sequence number and the log number. Then, based on this mapping relationship, the log number corresponding to the data to be submitted is determined, and this log number is used as a snapshot point for data replay.

[0060] Optionally, in this embodiment of the application, the Raft module can implement snapshot operations through copy technology (such as Copy-On-Write technology).

[0061] Optionally, when a new data node is added or other scenarios require the use of snapshots, the Raft protocol will copy the snapshot data and replay the logs after the snapshot point to obtain a complete set of data. That is, by using the snapshot and the logs stored in the Raft protocol, a copy of the same data is made to meet the requirements of new data node additions or other scenarios that require the use of snapshots.

[0062] 102. Determine the maximum sequence number submitted by the memory table, and query the log number corresponding to the maximum sequence number from the mapping queue;

[0063] In this step, a memory table is a data storage structure in memory. Each memory table can store a certain amount of data in a contiguous manner, such as 100 data items in a memory table.

[0064] In this embodiment, to reduce the time spent during restart, the maximum sequence number submitted by the memory table is determined. Each data change event corresponds to a sequence number, and the sequence numbers are incremented. The maximum sequence number in the memory table is the latest updated record. Therefore, the log point corresponding to this latest updated record can be sent to the Raft module. The Raft module uses this log point as a snapshot point and compensates for data in memory that has not yet been persisted to disk. Disk persistence refers to storing the data that needs to be stored on the local hard drive.

[0065] Optionally, a snapshot can be taken proactively when the program closes to store all data on the local disk, thus avoiding data loss and eliminating the need to replay logs upon restart.

[0066] 103. Store the combination of the log number and the maximum sequence number in the data file corresponding to the memory table;

[0067] In this embodiment of the application, the memory table is a structure that stores records in memory. When the memory table reaches a certain size or the amount of data it contains reaches a preset threshold, the memory table is frozen and the data contained in the memory table is written to a data file.

[0068] 104. In the event of a fault restart, traverse the multiple data files contained in memory and send the largest log number to the processing module; the largest log number is used by the processing module as a snapshot point and data replay is performed based on the snapshot point.

[0069] In this step, each data file contains multiple records, and each record contains a corresponding sequence number. By traversing each record of each data file in memory, the maximum sequence number is determined, and the log number corresponding to the maximum sequence number is the maximum log number (since both the sequence number and the log number are incrementing).

[0070] In this embodiment, the largest log number is sent to the Raft module, which uses this log point as a snapshot point and compensates for the data in memory that has not yet been persisted to disk.

[0071] Figure 2 A flowchart of a data processing method provided in an embodiment of this application is shown below. Figure 2 As shown, the method includes:

[0072] 201. Determine the data to be submitted and the log number corresponding to the data to be submitted. The log number is determined by the Raft module based on the data submitted in the past.

[0073] For example, taking the incrementing of the serial number as an example, if the current maximum serial number of the memory is 100, if new data is to be stored, a new serial number is generated, which can be 101 or other values.

[0074] It should be noted that this application is mainly used to write data in memory into the Raft module. If the program crashes and the data in memory is lost, it is necessary to recover this data, which is the data to be submitted.

[0075] Specifically, when a client sends a business request to the business system, the business system generates a record based on the request. Since the memory may not have had time to store the record yet, the data contained in the record needs to be stored as data to be submitted and its corresponding log number.

[0076] 202. Store the data to be submitted in memory, and determine the sequence number corresponding to the data to be submitted based on the historically stored data;

[0077] 203. Establish a mapping relationship between the log number and the sequence number, and form a mapping relationship queue composed of multiple mapping relationships. The mapping relationship queue contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory.

[0078] 204. Determine the maximum sequence number submitted by the memory table, and query the log number corresponding to the maximum sequence number from the mapping queue;

[0079] In this step, the memory table can store multiple data records, and each data record corresponds to a sequence number. The sequence numbers of the multiple data records are in ascending order according to the storage time. Therefore, the sequence number corresponding to the latest data stored in the memory table can be the maximum sequence number.

[0080] Furthermore, if the data is also idempotent, the mapping relationship can be selectively maintained every certain number of Raft logs, which can greatly reduce the number of mapping relationships contained in the mapping relationship queue.

[0081] Furthermore, it also includes: if the data contained in the memory table exceeds a preset threshold, storing the remaining acquired data into another memory table.

[0082] In this step, multiple memory tables may be included in memory. A preset threshold can be set as needed; for example, the preset threshold may be 100. If the amount of data contained in a memory table exceeds 100, the remaining data will be stored in another memory table.

[0083] 205. If the memory table is persisted to the hard disk, store the combination of the log number and the maximum sequence number in the data file corresponding to the memory table;

[0084] In this embodiment, correctly storing the Raft log number in the data file corresponding to the LSM-Tree storage engine can avoid consistency issues caused by storing the corresponding data in other locations. For example, if the data is written to other files, the memory table cannot be correctly persisted to the hard disk.

[0085] The purpose of persisting the memory table to the hard drive is to store the memory table on the local hard drive (disk) to avoid the problem of memory table loss when the program is closed and reopened if the memory table is not stored on the local hard drive.

[0086] 206. In the event of a fault restart, traverse the multiple data files contained in memory and send the largest log number to the processing module; the largest log number is used by the processing module as a snapshot point and data replay is performed based on the snapshot point.

[0087] In this step, a fault restart may result in data loss, so it is necessary to replay the data to ensure data consistency.

[0088] In this embodiment, during a fault restart, all data files in memory are traversed to obtain several Raft log numbers, with the largest Raft log number being the actual snapshot point. This log number is then transmitted to the Raft protocol, allowing log replay from the snapshot point.

[0089] In this embodiment of the application, the above-described solution can greatly reduce the time and resource overhead during fault restart, thereby also reducing the number of snapshots within a cycle.

[0090] For example, assuming that snapshots are taken every few log entries, the corresponding data volume for the LSM-Tree storage engine is the data volume of N in-memory tables. Therefore, the theoretical expectation for playback during a failure restart is approximately the data volume of N / 2 in-memory tables.

[0091] The optimized solution requires at most the number of records from one in-memory table to be replayed, with a theoretical expectation of 0.5 times the number of records from one in-memory table. If a less costly solution is used, recording the mapping relationship once every M records, then the theoretical expectation is that an additional M / 2 records are needed, where M is generally much smaller than the number of records in one in-memory table.

[0092] The optimized solution's cost during fault restarts is no longer dependent on the snapshot points of the latest Raft protocol, eliminating the need for more frequent snapshot generation. The replay log entries for a 0.5-memory table with M / 2 records are theoretically significantly fewer than those for N / 2-memory tables.

[0093] Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, as shown below. Figure 3 As shown, the device includes:

[0094] The first determining module 31 is used to determine the mapping relationship queue, which contains the mapping relationship between the log number submitted by the processing module and the sequence number submitted by the memory.

[0095] The second determining module 32 is used to determine the maximum sequence number submitted by the memory table and query the log number corresponding to the maximum sequence number from the mapping relationship queue;

[0096] Storage module 33 is used to store the combination of the log number and the maximum sequence number into the data file corresponding to the memory table;

[0097] The processing module 34 is used to traverse multiple data files contained in memory in the event of a fault restart and send the largest log number to the processing module; the largest log number is used by the processing module as a snapshot point and to perform data replay based on the snapshot point.

[0098] Optionally, in this embodiment of the application, the processing module may include a Raft module, and the first determining module 31 is specifically used to determine the data to be submitted and the log number corresponding to the data to be submitted, wherein the log number is determined by the Raft module based on the historically submitted data; store the data to be submitted in memory, and determine the sequence number corresponding to the data to be submitted based on the historically stored data; establish a mapping relationship between the log number and the sequence number, and form a mapping relationship queue by multiple mapping relationships.

[0099] Optionally, in this embodiment of the application, the second determining module 32 is specifically used to determine that the sequence number corresponding to the latest stored data arranged in chronological order in the memory table is the maximum sequence number, and to query the log number corresponding to the maximum sequence number from the mapping relationship queue.

[0100] Optionally, in this embodiment of the application, the storage module 33 is specifically used to store the combination of the log number and the maximum sequence number into the data file corresponding to the memory table when the memory table is persisted to the hard disk.

[0101] Optionally, in this embodiment of the application, the storage module 33 is further configured to store the remaining data obtained into another memory table when the data contained in the memory table exceeds a preset threshold.

[0102] Figure 3 The data processing device can perform Figure 2 The implementation principle and technical effects of the data processing method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the data processing device in the above embodiments performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0103] In one possible design, Figure 3 The data processing apparatus of the illustrated embodiment can be implemented as an electronic device, such as in actual applications. Figure 4 As shown, the electronic device may include a storage component 401 and a processing component 402;

[0104] The storage component 401 stores one or more computer instructions, wherein one or more computer instructions are invoked and executed by the processing component 402 to achieve... Figure 2 The data processing method described in the embodiments.

[0105] The processing component 402 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), refresh controllers, micro-refresh controllers, microprocessors, or other electronic components to perform the above-described actions. Figure 2 The data processing method described in the embodiments.

[0106] Storage component 401 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0107] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 2 The data processing method of the embodiment shown.

[0108] This application also provides a computer program product that, when run on a computer's processor, causes the computer's processor to execute the above-described... Figure 2 The data processing method of the embodiment shown.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: determining a mapping relationship queue, wherein the mapping relationship queue comprises a mapping relationship between a log number submitted by a processing module and a sequence number submitted by a memory table; the log number is determined by a log generated by the processing module for each data node generated data change event, and the memory table stores a certain amount of continuous data, and each data change event corresponds to a sequence number; determining the maximum sequence number submitted by the memory table, and querying the log number corresponding to the maximum sequence number from the mapping relationship queue; storing the combination of the log number and the maximum sequence number into the data file corresponding to the memory table; the data file comprises a plurality of records, each record comprises a corresponding sequence number, and the sequence number and the log number are both incremented; in the case of fault restart, each record of each data file contained in the memory is traversed, the maximum sequence number and the maximum log number corresponding to the maximum sequence number are determined, and the maximum log number is sent to the processing module; the maximum log number is used for the processing module to take the maximum log number as a snapshot point, and to play back data according to the snapshot point.

2. The method of claim 1, wherein, The processing module comprises a Raft module, and the mapping relationship queue is determined, comprising: determining the data to be submitted and the log number corresponding to the data to be submitted, wherein the log number is determined by the Raft module according to the historical submitted data; storing the data to be submitted into the memory, and determining the sequence number corresponding to the data to be submitted according to the historical stored data; establishing a mapping relationship between the log number and the sequence number, and forming a mapping relationship queue by a plurality of mapping relationships.

3. The method of claim 1, wherein, The maximum sequence number submitted by the memory table is determined, comprising: determining that the sequence number corresponding to the latest stored data arranged in time sequence in the memory table is the maximum sequence number.

4. The method of claim 1, wherein, The combination of the log number and the maximum sequence number is stored into the data file corresponding to the memory table, comprising: in the case of persisting the memory table to the hard disk, the combination of the log number and the maximum sequence number is stored into the data file corresponding to the memory table.

5. A data processing apparatus, characterized by, The method comprises the following steps: a first determining module is configured to determine a mapping relationship queue, wherein the mapping relationship queue comprises a mapping relationship between a log number submitted by a processing module and a sequence number submitted by a memory table; the log number is determined by a log generated by the processing module for each data node generated data change event, and the memory table stores a certain amount of continuous data, and each data change event corresponds to a sequence number; a second determining module is configured to determine the maximum sequence number submitted by the memory table, and query the log number corresponding to the maximum sequence number from the mapping relationship queue; a storage module is configured to store the combination of the log number and the maximum sequence number into the data file corresponding to the memory table; the data file comprises a plurality of records, each record comprises a corresponding sequence number, and the sequence number and the log number are both incremented; The processing module is configured to, in the case of a fault restart, traverse each record of each data file in a plurality of data files contained in the memory, determine a maximum sequence number and a maximum log number corresponding to the maximum sequence number, and send the maximum log number to the processing module; the maximum log number is used by the processing module as a snapshot point, and data playback is performed according to the snapshot point.

6. The apparatus of claim 5, wherein, The processing module includes a Raft module, and the first determining module is specifically configured to determine to-be-committed data and a log number corresponding to the to-be-committed data, the log number being determined by the Raft module according to historical committed data; store the to-be-committed data to the memory, and determine a sequence number corresponding to the to-be-committed data according to historical stored data; A mapping relationship between the log number and the sequence number is established, and a mapping relationship queue is formed by a plurality of mapping relationships.

7. The apparatus of claim 5, wherein, The second determining module is specifically configured to determine that a sequence number corresponding to the latest stored data arranged in a time sequence in the memory table is a maximum sequence number, and query a log number corresponding to the maximum sequence number from the mapping relationship queue.

8. The apparatus of claim 5, wherein, The storage module is specifically configured to, in the case of persisting the memory table to a hard disk, store a combination of the log number and the maximum sequence number to a data file corresponding to the memory table.

9. The apparatus of claim 7, wherein, The storage module is further configured to, in the case that data contained in the memory table exceeds a preset threshold, store the obtained remaining data to another memory table.

10. An electronic device, comprising: The computer program is stored in the computer storage medium, and when the computer program is executed by the computer, the computer program is used to implement the data processing method according to any one of claims 1-4.

11. A computer storage medium, characterized in that The computer program is stored in the computer storage medium, and when the computer program is executed by the computer, the computer program is used to implement the data processing method according to any one of claims 1-4.

12. A computer program product, characterised in that, When the computer program is executed on the processor of the computer, the computer program is used to enable the processor of the computer to execute the data processing method according to any one of claims 1-4.

Images