A data processing method and corresponding apparatus

Abstract

A data processing method applicable to database systems or distributed storage systems, particularly suitable for systems with high memory usage, such as in-memory databases, is disclosed. The method includes: during a snapshot period, writing at least one first object from memory to a snapshot file; wherein the snapshot file includes a first version number and first data content for each first object; writing a second version number and second data content of a second object corresponding to a write request within the snapshot period to a log file; wherein the second object is contained within at least one first object; and wherein the first version number of the first object in the snapshot file and the second version number of the second object in the log file are used to restore the data corresponding to the memory object. Because generating a snapshot does not require blocking write requests, database access latency is reduced. Furthermore, restoring data by comparing the current version number ensures data consistency after restoration.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a data processing method and corresponding apparatus. Background Technology

[0002] While in-memory databases primarily rely on memory access, persistence is also a key feature for high reliability. Persistent data in in-memory databases is typically used for data recovery after a failure or for master-slave synchronization. In-memory databases usually create persistent snapshots periodically; that is, at a selected point in time, all in-memory data prior to that point is fully scanned and then written to persistent storage.

[0003] During the snapshot generation period, write requests to memory data are typically blocked, meaning that write requests cannot be processed during snapshot generation, leading to significant access latency. If write requests are not blocked, memory data can be modified during snapshot generation, potentially causing inconsistencies between persistent storage and actual memory data, making it impossible to recover the memory data during memory crash recovery.

[0004] It is evident that existing snapshot generation solutions all introduce significant database access latency, impacting database performance. Summary of the Invention

[0005] This application provides a data processing method for reducing database access latency caused by snapshot generation. This application also provides corresponding apparatus, computer-readable storage media, and computer program products.

[0006] The first aspect of this application provides a data processing method, comprising: writing at least one first object in memory to a snapshot file during a first time period; wherein the snapshot file includes a first version number and a first data content of each first object; writing a second version number and a second data content of a second object corresponding to a write request during the first time period to a log file; wherein the second object is contained within at least one first object; wherein the first version number of the first object in the snapshot file and the second version number of the second object in the log file are used to restore the data corresponding to memory.

[0007] In this application, the first time period refers to the snapshot time period, which is the period from the snapshot start time to the end after a full scan of every object in memory and writing each object to the snapshot file. The snapshot start time is usually set periodically, and the snapshot end time is related to the time it takes to scan the objects in memory.

[0008] In this application, "writing" can be understood as storing, such as storing a snapshot file, storing a log file, or storing in memory.

[0009] In this application, at least one first object includes an object that already exists in memory at the snapshot start time, and an object newly added to memory before the snapshot end time.

[0010] In this application, a snapshot file refers to a persistent storage medium or a file within a persistent storage medium, which may include a disk, a solid-state drive (SSD), a distributed file system, or cloud storage, etc.

[0011] In this application, version numbers are used to mark different versions of an object. Typically, when an object is first written, its version number is initialized. Version numbers are usually marked in ascending order; for example, the initial version number is v = 1. Then, each time the object is updated (modified or deleted), the version number increments by 1; for example, when the object is updated for the first time, the version number changes from v = 1 to v = 2. For the same object, different version numbers correspond to different data contents. In memory, for the same object, the maximum version number and its corresponding data content are typically stored.

[0012] In this application, the objects written to the snapshot file are uniformly referred to as the first object. Regardless of the value, the version number of the first object is uniformly represented by the first version number, and the corresponding data content is uniformly represented by the first data content.

[0013] In this application, write requests include requests to add new objects, requests to modify the data of existing objects, or requests to delete data. It can also be understood that any request that involves data changes can be a write request.

[0014] In this application, the second object can be either the first object that already exists in memory at the snapshot start time, or the first object newly added to memory before the snapshot end time. The second object can be a part or all of the first object, and the second object is related to the write request. For example, if there are 5 first objects in memory during the snapshot period (the first period), denoted as object 1, object 2, object 3, object 4, and object 5 respectively, and a write request occurs for object 2 and object 3 during the snapshot period, then object 2 and object 3 can both be called second objects.

[0015] In this application, a log file refers to a storage medium used to store logs, or a file stored on that storage medium. The log file stores the version number, operation, and corresponding data content for each write request related to each object. However, log files are periodically deleted; therefore, data in memory cannot be recovered solely from log files.

[0016] In this application, all objects written to the log file are uniformly referred to as "second objects." Regardless of their numerical value, the version number of a second object is uniformly represented by the "second version number," and its corresponding data content is uniformly represented by the "second data content." The version numbers of objects in the log file and snapshot file may be the same, and the data content of objects in the log file and snapshot file may overlap.

[0017] In this application, the time outside the snapshot period can be referred to as the non-snapshot time. During the non-snapshot time, the version number and data content of the object corresponding to the write request will also be written to the log file.

[0018] In this application, snapshot files and log files are used to restore data corresponding to memory. This can be used to restore data in memory when memory crashes, or in various scenarios where data needs to be restored, such as data migration in a distributed scenario or data backup in a master-slave scenario. By using snapshot files and log files, the data corresponding to the same object can be restored by comparing the first version number and the second version number of the same object.

[0019] In the first aspect mentioned above, during the snapshot period, both the first version number and first data content of the first object in memory are written to the snapshot file, and the second version number and second data of the second object corresponding to the write request are written to the log file. When data recovery is needed, the snapshot file and the log file are used together to recover the data. In this way, writing requests are not blocked when generating a snapshot, and read / write requests can be processed normally, reducing database access latency. In addition, during data recovery, the snapshot file and the log file can be used to recover the data corresponding to the same object by comparing the first version number and the second version number, which can ensure the consistency of the recovered data.

[0020] In one possible implementation, writing at least one first object in memory to a snapshot file includes: for a first object whose length is less than a first threshold, writing the first version number and first data content of the first object to the snapshot file at once; for a first object whose length is greater than the first threshold, writing the first version number of the first object to the snapshot file at once, and writing the first data content of the first object to the snapshot file multiple times.

[0021] In this application, the first threshold can be pre-configured or dynamically set, and the first threshold can be adjusted according to needs.

[0022] In this application, a first object with a length less than a first threshold can be called a small object, and a first object with a length greater than the first threshold can be called a large object.

[0023] In this possible implementation, for small objects, a single write to the snapshot file can be completed quickly using fewer processor and input / output (I / O) resources. This allows for fast writing of small objects without affecting other database operations. For large objects, the large objects are split and written to the snapshot file in multiple parts. This avoids consuming processor and I / O resources for extended periods, allowing for writing of large objects without impacting other database operations.

[0024] In one possible implementation, if the length of the first object is greater than a first threshold, the first object includes multiple sub-objects; the first data content of the first object is written to the snapshot file in multiple parts, including:

[0025] Multiple sub-objects are written to the snapshot file in multiple batches; each batch writes at least one sub-object, and each sub-object has at least one identifier, which is used to restore the data content corresponding to at least one sub-object.

[0026] In this application, at least one sub-object has at least one identifier. Each sub-object may have a unique identifier, or multiple sub-objects may share a single unique identifier.

[0027] In this application, the identifier of a sub-object can be either inherent to the sub-object or assigned to a sub-object of the first object when the first object is written into memory.

[0028] In this application, the first object with its own sub-object identifiers typically has the following characteristics: it includes sub-objects; each sub-object has a unique identifier; update / delete operations on sub-objects are reentrant (i.e., after an operation is executed, the data content will not change due to repeated execution of the operation); the structure of such first objects typically includes hash tables, mapping tables, sets, sorted sets, skip lists, etc. For example, if the set {1, 2, 3} includes three sub-objects, and a delete operation is received on sub-object 2 in the set, receiving another delete operation on sub-object 2 after sub-object 2 has been deleted will not change the set.

[0029] In this application, the first object that needs to allocate identifiers for sub-objects when writing to memory typically has the following characteristics: it includes sub-objects; the sub-objects themselves do not have unique identifiers; update / delete operations on sub-objects are not reentrant (i.e., after an operation is executed, the data content will change because the operation is executed repeatedly); there are sequential or relative relationships between different sub-objects in this type of first object, such as: for a linked list structure object, for the operation of appending an element to the tail of the linked list, if the same append operation is executed repeatedly, the length of the linked list will increase successively.

[0030] In this possible implementation, large objects with a length greater than the first threshold are divided into sub-objects and written to the snapshot file in multiple batches. The corresponding data can be marked by the sub-object identifier, which is beneficial for managing data written in different batches and also for restoring according to the sub-object identifier.

[0031] In one possible implementation, if the child objects of the first object do not have at least one identifier, the method further includes: when writing the first object to memory, allocating at least one identifier for the child objects of the first object; wherein each child object corresponds to one identifier.

[0032] In this possible implementation, by assigning a unique identifier to the child object that does not have an identifier, it is easier to write the first object to the snapshot file or log file later, and the consistency of the data after recovery can also be guaranteed.

[0033] In one possible implementation, when writing the second version number and second data content of the second object corresponding to the write request in the first time period to the log file, the method further includes: if the write request is an insertion request, modification request or deletion request for the target sub-object, associating the identifier of the predecessor sub-object adjacent to the target sub-object and the identifier of the successor sub-object with the identifier of the target sub-object and writing them to the log file.

[0034] In this application, the predecessor sub-object, the target sub-object, and the successor sub-object are sub-objects arranged in sequence. The predecessor sub-object is the neighbor preceding the target sub-object, and the successor sub-object is the neighbor following the target sub-object.

[0035] In this possible implementation, because the large objects in this application that do not have identifiers for sub-objects are not reentrant, and because the data between sub-objects has sequential or relative relationships, when writing the data corresponding to the target sub-object to the log file, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object are also associated and written to the log file. This ensures data consistency during subsequent data recovery.

[0036] In one possible implementation, if the length of the first object is less than a first threshold, the data corresponding to the memory is restored, including: restoring the first version number and the first data content of the first object in the snapshot file to memory; if the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the restored data content, and the restored data content is marked with the second version number; if the second version number of the first object in the log file is not greater than the first version number, then the second version number and the second data content of the first object restored from the log file are discarded.

[0037] In this application, the second data content can be an operation on the first object, or an operation plus data content.

[0038] Applying the second data content to the first data content to obtain the restored data content can be divided into several scenarios: First, if the operation corresponding to the second data content is an insertion operation, then the first data content is merged with the data content corresponding to the insertion operation; second, if the operation corresponding to the second data content is a modification operation, then the data content corresponding to the modification operation is used to replace the corresponding data in the first data content; third, if the operation corresponding to the second data content is a deletion operation, then the data content corresponding to the deletion operation is deleted from the first data content; fourth, if the operation corresponding to the second data content is a copy operation, then the data content corresponding to the second version number is the same as the first data content, and either the first data content or the data content corresponding to the copy operation is deleted.

[0039] In this possible implementation, the first version number and first data content of the first object in the snapshot file are first restored to memory. Then, the first version number in memory is compared with the second version number to determine if the restored first version number is greater than the second version number. If the second version number is greater than the first version number, the second data content needs to be applied to the first data content. This ensures that the data restored to memory is consistent with the data before the memory crash. If the second version number is not greater than the first version number, it means that the first version number is already a larger version number, and the second version number and second data content of the first object restored from the log file can be discarded.

[0040] In one possible implementation, if the length of the first object is greater than a first threshold and the first data contents of the first objects are not related to each other, the data corresponding to the memory is restored, including: restoring the first version number and the first data contents of the first object in the snapshot file to memory; if the second version number of the first object in the log file is greater than the first version number, then the second data contents are applied to the first data contents to obtain the restored data contents, and the restored data contents are marked with the second version number; if the second version number of the first object in the log file is not greater than the first version number, then the second version number and the second data contents of the first object restored from the log file are discarded.

[0041] In this application, "not related to each other" means that the data content of different sub-objects of the first object does not need to maintain a sequential or relative relationship. This type of data is the reentrant data described above.

[0042] In this possible implementation, for reentrant data, the data in memory can be recovered by comparing version numbers. This ensures that the data recovered to memory is consistent with the data before the memory crash.

[0043] In one possible implementation, if the length of the first object is greater than a first threshold and the first data contents of the first objects are interconnected, the data corresponding to the memory is restored, including: restoring the first version number and first data contents of the first object in the snapshot file to memory; and using or discarding the second version number and second data contents of the first object restored from the log file according to the first version number and second version number of the first object and the verification rules.

[0044] In this application, "interrelated" refers to the sequential or relative relationship between the data contents of different sub-objects of the first object. This type of data is the non-reentrant data described above.

[0045] In this possible implementation, for data that is not reentrant, because there are sequential or relative relationships between the data of different sub-objects, it is necessary not only to compare the version numbers but also to check whether the first object meets the verification rules in order to determine whether to use the log file or discard the second version number and second data content of the first object recovered from the log file. This ensures that when restoring data, the target sub-object is restored to the correct location, and that the data of the restored first object is consistent with the data before the memory crash.

[0046] In one possible implementation, if the second version number of the first object is not greater than the first version number, or if the first object does not meet the verification rules, then the second version number and the second data content of the first object recovered from the log file are discarded.

[0047] In this possible implementation, if the second version number of the first object is not greater than the first version number, it means that the second version number and the second data content of the first object in the log file are no longer needed, and the second version number and the second data content of the first object recovered from the log file are discarded. If the first object does not meet the verification rules, it means that the data content corresponding to the target sub-object in the log file does not need to be restored to memory and can be discarded directly.

[0048] In one possible implementation, if the second version number of the first object is greater than the first version number and the first object meets the verification rules, then the second data content is applied to the first data content to obtain the restored data content, and the restored data content is marked with the second version number.

[0049] In this possible implementation, if the second version number of the first object is greater than the first version number, restoring the second version number and the second data content to memory can ensure consistency with the memory data before the crash.

[0050] In one possible implementation, satisfying the verification rules includes at least one of the following:

[0051] If the predecessor child object of the target child object in the log file is used to find the predecessor child object in memory, and the successor child object is found using the successor child object, both will succeed.

[0052] If the target sub-object in the log file corresponds to the insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, and both the predecessor sub-object and the successor sub-object are adjacent to the target sub-object;

[0053] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object must satisfy an order relationship.

[0054] This possible implementation provides multiple scenarios that satisfy the verification rules, which helps ensure the consistency of the recovered data.

[0055] In one possible implementation, the validation rule not being satisfied includes at least one of the following:

[0056] If using the identifier of the predecessor sub-object of the target sub-object in the log file to find the predecessor sub-object in memory fails, or using the identifier of the successor sub-object to find the successor sub-object fails;

[0057] If the target sub-object in the log file corresponds to an insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, but the predecessor sub-object and the successor sub-object are not adjacent to the target sub-object;

[0058] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object do not satisfy the order relationship.

[0059] This possible implementation provides multiple scenarios where the verification rules are not met, which helps ensure the consistency of the recovered data.

[0060] A second aspect of this application provides a data processing apparatus, comprising:

[0061] The first processing unit is configured to write at least one first object in memory into a snapshot file during a first time period; wherein the snapshot file includes a first version number and first data content for each first object;

[0062] The second processing unit is used to write the second version number and second data content of the second object corresponding to the write request within the first time period into the log file; wherein the second object is contained in at least one first object;

[0063] The snapshot file and log file are used to restore the data corresponding to the first object.

[0064] In one possible implementation, the first processing unit is specifically used for:

[0065] For at least one first object whose length is less than a first threshold, the first version number and first data content of the first object are written to the snapshot file at once;

[0066] For at least one first object whose length is greater than a first threshold, the first version number of the first object is written to the snapshot file once, and the first data content of the first object is written to the snapshot file multiple times.

[0067] In one possible implementation, the first processing unit is specifically used to: if the length of the first object is greater than a first threshold, the first object includes multiple sub-objects; write the multiple sub-objects to the snapshot file in multiple batches; wherein, at least one sub-object is written each time, and at least one sub-object has at least one identifier, and at least one identifier is used to recover the data content corresponding to at least one sub-object.

[0068] In one possible implementation, the first processing unit is further configured to: if the child objects of the first object do not have at least one identifier, when writing the first object to memory, allocate at least one identifier to the child objects of the first object; wherein each child object corresponds to one identifier.

[0069] In one possible implementation, the second processing unit is further configured to, if the write request is an insert request, modification request, or deletion request for the target sub-object, associate the identifiers of the predecessor sub-objects adjacent to the target sub-object and the identifiers of the successor sub-objects with the identifier of the target sub-object and write them to a log file.

[0070] In one possible implementation, the device further includes a third processing unit.

[0071] The third processing unit is used to handle situations where the length of the first object is less than the first threshold:

[0072] Restore the first version number and first data content of the first object in the snapshot file to memory;

[0073] If the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number.

[0074] If the second version number of the first object in the log file is not greater than the first version number, then discard the second version number and the second data content of the first object recovered from the log file.

[0075] In one possible implementation, the device further includes a third processing unit.

[0076] The third processing unit is configured to handle situations where the length of the first object is greater than a first threshold and the first data contents of the first objects are not related to each other:

[0077] Restore the first version number and first data content of the first object in the snapshot file to memory;

[0078] If the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number.

[0079] If the second version number of the first object in the log file is not greater than the first version number, then discard the second version number and the second data content of the first object recovered from the log file.

[0080] In one possible implementation, the data processing device further includes a third processing unit.

[0081] The third processing unit is configured to handle cases where the length of the first object is greater than a first threshold, and the first data contents of the first objects are correlated:

[0082] Restore the first version number and first data content of the first object in the snapshot file to memory;

[0083] Based on the first and second version numbers of the first object, and the verification rules, use the log file or discard the log file.

[0084] In one possible implementation, the third processing unit is used to discard the second version number and the second data content of the first object recovered from the log file if the second version number of the first object is not greater than the first version number, or if the first object does not meet the verification rules.

[0085] In one possible implementation, the third processing unit is configured to apply the second data content to the first data content if the second version number of the first object is greater than the first version number and the first object satisfies the verification rules, thereby obtaining the restored data content, and marking the restored data content with the second version number.

[0086] In one possible implementation, satisfying the verification rules includes at least one of the following:

[0087] If the predecessor child object of the target child object in the log file is used to find the predecessor child object in memory, and the successor child object is found using the successor child object, both will succeed.

[0088] If the target sub-object in the log file corresponds to the insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, and both the predecessor sub-object and the successor sub-object are adjacent to the target sub-object;

[0089] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object must satisfy an order relationship.

[0090] In one possible implementation, the validation rule not being satisfied includes at least one of the following:

[0091] If using the identifier of the predecessor sub-object of the target sub-object in the log file to find the predecessor sub-object in memory fails, or using the identifier of the successor sub-object to find the successor sub-object fails;

[0092] If the target sub-object in the log file corresponds to an insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, but the predecessor sub-object and the successor sub-object are not adjacent to the target sub-object;

[0093] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object do not satisfy the order relationship.

[0094] The first processing unit, the second processing unit, and the third processing unit mentioned above can also be implemented by a single processing unit.

[0095] A third aspect of this application provides a data processing apparatus. The apparatus may include at least one processor, a memory, and a communication interface. The processor is coupled to the memory and the communication interface. The memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other devices under the control of the processor. When executed by the processor, the instructions cause the processor to perform the methods of the first aspect or any possible implementation thereof.

[0096] The fourth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0097] The fifth aspect of this application provides a computer program product that stores one or more computer execution instructions, wherein when the computer execution instructions are executed by a processor, the processor executes the method described in the first aspect or any possible implementation thereof.

[0098] A sixth aspect of this application provides a chip system including a processor for supporting a computer device in implementing the functions described in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0099] The seventh aspect of this application provides a database system including at least one data node, wherein the data node is used to perform the method described in the first aspect or any possible implementation thereof.

[0100] The technical effects of the second aspect or any possible implementation of the second aspect, as well as the technical effects of the third to seventh aspects, can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0101] Figure 1A This is a schematic diagram of the architecture of a database system;

[0102] Figure 1B This is a schematic diagram of the architecture of a distributed database system provided in an embodiment of this application;

[0103] Figure 1C This is another schematic diagram of the architecture of the distributed database system provided in the embodiments of this application;

[0104] Figure 2A This is a schematic diagram of the architecture of a general-purpose computer system provided in an embodiment of this application;

[0105] Figure 2B This is another schematic diagram of the architecture of the general-purpose computer system provided in the embodiments of this application;

[0106] Figure 3A This is a schematic diagram illustrating an example of the data relationship between snapshot files and log files provided in an embodiment of this application;

[0107] Figure 3B This is another example of the data relationship between snapshot files and log files provided in the embodiments of this application;

[0108] Figure 4 This is a schematic diagram of an embodiment of the data processing method provided in this application;

[0109] Figure 5A This is an example schematic diagram of a snapshot time period provided in an embodiment of this application;

[0110] Figure 5B This is an example schematic diagram of the first object within the snapshot time period provided in the embodiments of this application;

[0111] Figure 5C This is an example schematic diagram illustrating the classification of the first object provided in an embodiment of this application;

[0112] Figure 6A This is a schematic diagram of another embodiment of the data processing method provided in this application;

[0113] Figure 6B This is an example schematic diagram of the data processing method provided in the embodiments of this application;

[0114] Figure 7A This is a schematic diagram of another embodiment of the data processing method provided in this application;

[0115] Figure 7B This is an example diagram illustrating the multiple writing of a large object to a snapshot file according to an embodiment of this application;

[0116] Figure 8A This is a schematic diagram of another embodiment of the data processing method provided in this application;

[0117] Figures 8B to 8D These are several example schematic diagrams provided in the embodiments of this application;

[0118] Figure 9A This is a schematic diagram of a single-machine system provided in an embodiment of this application;

[0119] Figure 9B This is a schematic diagram of a scenario for partition data migration in a distributed cluster provided in an embodiment of this application;

[0120] Figure 10A This is a schematic diagram of a data replication scenario under a primary / standby deployment provided in the embodiments of this application;

[0121] Figure 10B This is a schematic diagram of a scenario of a distributed system based on a local disk provided in an embodiment of this application;

[0122] Figure 11 This is a schematic diagram of the structure of the data processing apparatus provided in an embodiment of this application;

[0123] Figure 12 This is another structural schematic diagram of the data processing apparatus provided in the embodiments of this application;

[0124] Figure 13 This is another schematic diagram of the cluster database system provided in the embodiments of this application. Detailed Implementation

[0125] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will understand, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0126] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0127] This application provides a data processing method to reduce database access latency caused by snapshot generation. This application also provides corresponding apparatus, computer-readable storage media, and computer program products. These are described in detail below.

[0128] The method provided in this application embodiment can be applied to a database system. Figure 1A This illustrates a typical logical architecture of a database system, based on Figure 1A Database system 100 includes database 110 and database management system (DBMS) 130.

[0129] Database 110 is an organized collection of data stored in data storage 120; that is, a related collection of data organized, stored, and used according to a specific data model. Depending on the data model used to organize the data, the data can be divided into various types, such as relational data, graph data, and time series data. Relational data is data modeled using a relational model, usually represented as a table, where rows represent a set of related values ​​for an object or entity. Graph data, simply called a "graph," is used to represent relationships between objects or entities, such as social relationships. Time series data is a series of data columns recorded and indexed in chronological order, used to describe the state changes of an object over time.

[0130] The database management system 130 is the core of the database system, serving as system software for organizing, storing, and maintaining data. Clients 200 can access the database 110 through the database management system 130, and database administrators also use the database management system for database maintenance. The database management system 130 provides various functions for clients 200 to create, modify, and query the database; clients 200 can be applications or user devices. The functions provided by the database management system 130 may include, but are not limited to, the following: (1) Data definition function: The database management system 130 provides a data definition language (DDL) to define the structure of the database 110. The DDL is used to describe the database framework and can be stored in the data dictionary; (2) Data access function: The database management system 130 provides a data manipulation language (DML) to implement basic access operations on the database 110, such as retrieval, insertion, modification and deletion; (3) Database operation management function: The database management system 130 provides data control function to effectively control and manage the operation of the database 110 to ensure that the data is correct and valid; (4) Database creation and maintenance function: including loading the initial data of the database, database dumping, recovery, reorganization, system performance monitoring and analysis, etc.; (5) Database transmission: The database management system provides data transmission to realize communication between the client and the database management system, which is usually coordinated with the operating system.

[0131] Data storage 120 includes, but is not limited to, solid-state drives (SSDs), disk arrays, cloud storage, or other types of non-transitory computer-readable storage media. Those skilled in the art will understand that a database system may include more than […]. Figure 1A The components shown are fewer or more components, or include those with Figure 1A The components shown are different components. Figure 1A Only components more relevant to the implementation disclosed in the embodiments of the present invention are shown.

[0132] The database system provided in this application embodiment can be a distributed database system (DDBS). DDBS is a logically unified database composed of multiple physically dispersed database units connected by a computer network. Each connected database unit is called a site or node. The following describes the process in conjunction with... Figure 1B and Figure 1C Introducing DDBS.

[0133] Figure 1BThis diagram illustrates a distributed database system employing a shared-nothing architecture. The system includes one or more coordinator nodes (CNs) and multiple data nodes (DNs). Each DN can handle read or write requests, storing the maximum version number of an object and its corresponding data content in memory. Snapshot processing is performed during snapshot periods, writing the first version number and first data content of the first object in memory to a snapshot file, and writing the second version number and second data content of the second object corresponding to a write request to a log file.

[0134] It should be noted that a distributed database system may also include one or more GTMs (such as...). Figure 1B The system comprises a first GTM and a second GTM. The first GTM acts as the master GTM, while the second GTM backs up the data of the first GTM and takes over its operation when the first GTM fails, ensuring the high reliability of DDBS. CNs and DNs communicate via a network channel. Each DN has its own dedicated hardware resources (such as data storage), operating system, and database. CNs, DNs, the first GTM, or the second GTM communicate via a network channel. In one embodiment, this network channel can consist of network devices such as switches, routers, and gateways. CNs, DNs, and GTMs jointly implement the functions of the database management system, providing clients with services such as database querying (retrieval), insertion, modification, and deletion. In one embodiment, a database management system is deployed on each CN, DN, and GTM. In this distributed database system, data is allocated to various DNs according to the database model and application characteristics. Query tasks are divided into several parts by the CN and executed in parallel on multiple DNs, coordinating computation to provide database services as a whole. All communication functions can be implemented on a high-bandwidth network interconnection system.

[0135] In a distributed database system, the CN, DN, first GTM, or second GTM can be a physical machine, such as a database server, or a virtual machine (VM) or container running on abstract hardware resources. In one embodiment, the CN, DN, first GTM, or second GTM is a VM or container, and the network channel is a virtual switching network, which includes virtual switches. The database management system deployed in the CN, DN, first GTM, or second GTM is a DBMS instance, which can be a process or a thread. These DBMSs work together to complete the functions of the database relational system. In another embodiment, the CN, DN, first GTM, or second GTM is a physical machine, and the network channel includes one or more switches, which can be a storage area network (SAN) switch, an Ethernet switch, a fiber optic switch, or other physical switching equipment.

[0136] Figure 1C This is a schematic diagram of a distributed database system using a shared-storage architecture. Figure 1C In the distributed database system shown, a shared data storage device stores data that can be shared by multiple Data Nodes (DNs). DNs can perform read and write operations on the data in the data storage device via a network channel. The shared data storage device can be a shared disk array. The CNs, DNs, and the first or second GTM communicate via a network channel, which can be found in the above description. Figure 1B Please refer to the relevant introductions for understanding. In this system, data can be stored in a shared data storage device. Query tasks are divided into several parts by the CN (Network Controller), executed in parallel on multiple DNs (Data Networks), and compute collaboratively, providing database services as a whole. All communication functions can be implemented on a high-bandwidth interconnected network. (Similar to...) Figure 1B Similar to the shared-nothing architecture of the distributed database system described, CN, DN, first GTM, or second GTM can be either physical machines or virtual machines.

[0137] In all embodiments of this application, the data storage of the database system includes, but is not limited to, SSDs, disk arrays, or other types of non-transient computer-readable media. Figure 1A-1C Although a database is not shown, it should be understood that a database is stored in a data storage device. Those skilled in the art will understand that a database system may include more than [a certain number of data storage devices]. Figure 1A-1C The components shown are fewer or more components, or include those with Figure 1A-1C The components shown are different components. Figure 1A-1COnly components more relevant to the implementation disclosed in the embodiments of this application are shown. However, those skilled in the art will understand that a distributed database system can contain any number of CNs and DNs. The database management system functions of each CN and DN can be implemented by appropriate combinations of software, hardware, and / or firmware running on each CN and DN.

[0138] The above Figure 1B and Figure 1C The described distributed database system includes multiple Data Domains (DNs) and multiple Data Centers (CNs). DNs are responsible for executing specific execution plans. Aside from the different data they store, DNs are architecturally identical and support various data distribution methods. CNs can be elected from DNs using an election algorithm or can adopt a different architecture. Depending on the specific deployment of the distributed database, there can be one or more CNs. Database modules can be deployed equally across CNs and DNs, and their roles within the distributed database can be configured through configuration files.

[0139] The solutions provided in this application can be applied to various types of database systems or storage systems, especially to in-memory databases or storage systems that use a lot of memory.

[0140] To address the database access latency issue that arises when using blocking write requests to generate snapshots, existing technologies have proposed implementing snapshots through fork technology or multiple versions.

[0141] The fork technique for snapshotting involves creating a child process from the parent process using the fork function. The child process shares the same memory space as the parent, and writes all the data in its memory space to a persistent snapshot, thus completing the snapshot generation process. During snapshot generation, the parent process can handle read / write requests normally, resolving access latency issues. However, the fork technique requires copying the parent process's page table to the child process's page table. Since the parent process's virtual address space is typically large, this copying operation can cause the parent process to be blocked for extended periods, preventing it from processing business requests and resulting in significant latency jitter. Furthermore, if the parent process receives a large number of write requests during snapshot generation, frequent write operations can occur, leading to extensive physical memory copying and substantial memory consumption.

[0142] Multi-version snapshots involve maintaining multiple versions of data in memory to preserve a snapshot state at a specific point in time. Background persistent snapshot tasks are constructed based on this version, and new write requests are maintained in a new incremental table. This allows for data recovery during crash recovery using the snapshot of the version corresponding to a specific point in time and the data in the new incremental table. Multi-version snapshots also avoid blocking write requests, solving the access latency problem. However, if there are many data changes during snapshot generation, simultaneously saving both historical and changed versions can lead to significant memory overhead. Furthermore, maintaining multiple versions of data during snapshot generation requires searching through multiple versions of data before determining the data the user actually wants, increasing the average latency of data access.

[0143] As can be seen from the fork technique or the multi-version snapshot method mentioned above, although the problem of not needing to block write requests is solved, they all bring a lot of memory overhead and create new latency issues.

[0144] Based on this, the data processing method provided in this application embodiment can solve the problem of not needing to block write requests, and will not generate new latency problems or large memory overhead problems by solving the problem of blocking write requests.

[0145] The data processing method provided in this application can be applied to data nodes in the aforementioned database system. This data node can be a physical machine or a virtual machine, and both can be general-purpose computer systems. (See [link to relevant documentation] for details.) Figure 2A and Figure 2B To understand.

[0146] like Figure 2A As shown, this general-purpose computer system may include a software layer and a hardware layer. The software layer may include multiple functional programs, such as data read / write programs, memory object management programs, snapshot task management programs, persistent file management programs, and crash recovery programs. The hardware layer may include a processor, memory, persistent storage media, and communication interfaces. The persistent storage media may include a hard disk, a solid-state drive (SSD), a distributed file system, or cloud storage.

[0147] like Figure 2B As shown, memory can include one or more objects; persistent storage media include snapshot files and log files. Snapshot files and log files can be storage media used to store snapshots and logs respectively, or they can be files stored on the respective storage media.

[0148] Among them, the data read / write function program is used to handle data read / write requests from the upper-layer database application;

[0149] Memory object management function program: used to manage objects in memory;

[0150] Snapshot task management function program: used to handle tasks triggered periodically or in specific scenarios, responsible for a full scan of objects in memory and writing the data of the scanned objects to snapshot files in persistent storage media;

[0151] Persistent file management function program: used to manage log files in persistent storage media;

[0152] Crash recovery function: After a database process crashes and recovers, it is responsible for reading data from snapshot files and log files in persistent storage media, loading it into memory, and restoring objects in memory.

[0153] In this embodiment, the relationship between snapshot files and log files can be found in [reference needed]. Figure 3A or Figure 3B To understand.

[0154] like Figure 3A As shown, there is some overlap between the data in the snapshot file and the log file. This may occur after a snapshot has started, when a write request is made to an object that has already been scanned. The data corresponding to this write request will be written to the log file, but not stored in the snapshot file. Additionally, after a snapshot ends and before the next snapshot begins, data generated by write requests will also be written to the log file.

[0155] In this application, "writing" can be understood as storing, such as storing a snapshot file, storing a log file, or storing in memory.

[0156] like Figure 3B As shown, the log file is contained within the snapshot file. This situation may occur when no write requests occur after the snapshot begins, and no new data is generated. The existing data in the log file will be written to the snapshot file when the memory is fully scanned.

[0157] Based on the aforementioned database system or distributed storage system, this application provides a data processing method to reduce database access latency caused by snapshot generation.

[0158] like Figure 4 As shown, the data processing method provided in this application embodiment includes:

[0159] S401. In a first time period, at least one first object in memory is written to a snapshot file; wherein the snapshot file includes a first version number and a first data content for each first object.

[0160] In this application, the first time period refers to the snapshot time period, which is the period from the snapshot start time to the end after a full scan of every object in memory and writing each object to the snapshot file. The snapshot start time is usually set periodically, and the snapshot end time is related to the time taken to scan the objects in memory. Figure 5A As shown, on the timeline, snapshots begin at time point 1, and end at time point 5. The period from time point 3 to time point 5 is the snapshot time period.

[0161] In this application, at least one first object includes objects that already exist in memory at the snapshot start time, and objects newly added to memory before the snapshot end time. See [link to relevant documentation] for more information on first objects. Figure 5B To understand, such as Figure 5B As shown, at the snapshot start time (time point 3), the memory contains objects 1, 2, and 3. At time point 3, objects 1, 2, and 3 are all referred to as the first object. At the snapshot end time (time point 5), the memory contains objects 1, 2, 3, and 4. Object 4 is the object newly inserted in response to a write request during the snapshot period. At this time, objects 1, 2, 3, and 4 are also referred to as the first object. In this example, the first objects during the snapshot period include objects 1, 2, 3, and 4.

[0162] In this application, version numbers are used to mark different versions of an object. Typically, when an object is first written, its version number is initialized. Version numbers are usually marked in ascending order; for example, the initial version number is v = 1. Then, each time the object is updated (modified or deleted), the version number increments by 1; for example, when the object is updated for the first time, the version number changes from v = 1 to v = 2. For the same object, different version numbers correspond to different data contents. In memory, for the same object, the maximum version number and its corresponding data content are typically stored.

[0163] In this application, the objects written to the snapshot file are uniformly referred to as the first object. Regardless of the value, the version number of the first object is uniformly represented by the first version number, and the corresponding data content is uniformly represented by the first data content.

[0164] The first version number and first data content of the first object in the snapshot file can be understood by referring to Table 1. As shown in Table 1:

[0165] Table 1

[0166] First object Version number First data content Object 1 v2 D12 Object 2 v3 D23 Object 3 v2 D32 Object 4 v1 D41

[0167] It should be noted that the snapshot file is not limited to the contents of Table 1 above. The snapshot file may also include other objects, or the data content D11 of the v1 version of object 1, the data content D21 of the v1 version of object 2, the data content D22 of the v2 version of object 2, the data content D31 of the v1 version of object 3, etc.

[0168] S402. Write the second version number and second data content of the second object corresponding to the write request within the first time period to the log file; wherein the second object is contained in at least one first object.

[0169] In this application, write requests include requests to add new objects, requests to modify the data of existing objects, or requests to delete data. It can also be understood that any request that involves data changes can be a write request.

[0170] In this application, the second object can be either the first object that already exists in memory at the start time of the snapshot, or a newly added first object in memory before the end time of the snapshot. The second object can be a part or all of the first object, and it is related to write requests. For example, if there are 5 first objects in memory during the snapshot period (the first period), denoted as object 1, object 2, object 3, object 4, and object 5 respectively, and a write request occurs for object 2 and object 3 during the snapshot period, then object 2 and object 3 can both be considered second objects. Or, as... Figure 5B As shown, at the end of the snapshot (time point 5), there are four first objects in memory: object 1, object 2, object 3, and object 4. Among them, object 4 is the object newly inserted in response to the write request during the snapshot period, so object 4 is also called the second object.

[0171] For the example in Table 1 above, the log file should include at least the contents of Table 2, as shown in Table 2:

[0172] Second object Version 2 Second data content Object 4 v1 D41

[0173] Of course, Table 2 may also include version numbers and corresponding data content of other objects in Table 1, and this application does not limit this.

[0174] In this application, a log file refers to a storage medium used to store logs, or a file stored on that storage medium. The log file stores the version number and data content related to all write requests for each object; however, log files are deleted periodically, so recovering data from memory cannot be guaranteed solely through log files.

[0175] In this application, all objects written to the log file are uniformly referred to as "second objects." Regardless of their numerical value, the version number of a second object is uniformly represented by the "second version number," and its corresponding data content is uniformly represented by the "second data content." The version numbers of objects in the log file and snapshot file may be the same, and the data content of objects in the log file and snapshot file may overlap.

[0176] In this application, the time outside the snapshot period can be referred to as the non-snapshot time. During the non-snapshot time, the version number and data content of the object corresponding to the write request will also be written to the log file.

[0177] The first version number of the first object in the snapshot file and the second version number of the second object in the log file are used to restore the data corresponding to memory.

[0178] In this application, snapshot files and log files are used to restore data corresponding to memory. This can be used to restore data in memory when memory crashes, or in various scenarios where data needs to be restored, such as data migration in a distributed scenario or data backup in a master-slave scenario. By using snapshot files and log files, the data corresponding to the same object can be restored by comparing the first version number and the second version number of the same object.

[0179] In the solution provided in this application embodiment, during the snapshot period, both the first version number and first data content of the first object in memory are written to the snapshot file, and the second version number and second data of the second object corresponding to the write request are written to the log file. When data recovery is needed, the snapshot file and the log file are used together to recover the data. In this way, writing requests are not blocked when generating a snapshot, and read / write requests can be processed normally, reducing database access latency. Furthermore, during data recovery, the snapshot file and the log file can be used to recover the data corresponding to the same object by comparing the first version number and the second version number, ensuring data consistency after recovery. In addition, compared to the existing technology of implementing snapshots through fork technology or multiple versions, this method avoids memory overhead and latency issues.

[0180] In this embodiment, the first object can be divided into small objects and large objects according to the length of the object in memory; wherein, the first object with a length less than a first threshold is a small object, and the first object with a length greater than the first threshold is a large object. The first threshold can be pre-configured or dynamically set, and can be adjusted as needed. Figure 5C As shown, if the length of object 1 is less than the first threshold, then object 1 is a small object; if the length of object 2 is greater than the first threshold, then object 2 is a large object. A large object in this application may include multiple sub-objects, and sub-objects may or may not have identifiers.

[0181] Large objects can be further divided into two categories. Different types of large objects have different characteristics. In general, the first object in the embodiments of this application can also have three types, which will be introduced below.

[0182] Category 1: Small objects whose length is less than the first threshold;

[0183] The second category includes large objects whose length exceeds the first threshold, as well as sub-objects of the first object that have their own identifiers.

[0184] This type of large object may include the following characteristics:

[0185] 1. Includes sub-objects;

[0186] 2. Sub-objects have unique identifiers;

[0187] 3. Update / delete operations on sub-objects are reentrant (i.e., after an operation is executed, the data content will not change because the operation is executed repeatedly); for example, if the set {1, 2, 3} contains three sub-objects, if a delete operation is received on sub-object 2 in the set, and after sub-object 2 is deleted, receiving another delete operation on sub-object 2 will not change the set.

[0188] 4. The structures of this type of first object typically include hashes / mapping tables / sets / sorted sets / skip lists, etc.

[0189] The third category includes large objects whose length exceeds the first threshold, and child objects of the first object that do not have an identifier.

[0190] This type of large object may include the following characteristics:

[0191] 1. Includes sub-objects;

[0192] 2. The sub-object itself does not have a unique identifier;

[0193] 3. Update / delete operations on sub-objects are not reentrant (i.e., after an operation is executed, the data content will change because the operation is executed repeatedly); for example, for a linked list structure, if the same append operation is executed multiple times, the length of the linked list will increase each time.

[0194] 4. There are sequential or relative relationships between different sub-objects in this type of first object. For example, in a linked list structure, if the same append operation is repeatedly performed on the tail of the linked list, the length of the linked list will increase step by step.

[0195] Each type of first object involves several steps: writing the first object to memory and writing it to a log file, taking a snapshot of the first object in memory, and restoring the data corresponding to the memory object. The following describes the data processing procedures for each type of first object:

[0196] like Figure 6A As shown, the data processing procedure for the first object of the first type includes:

[0197] The steps for writing the first object into memory and writing it to the log file include:

[0198] S601. Write the first object into memory for the first time and assign an initial version number v1 to the first object.

[0199] S602. Write the version number v1 of the first object and the corresponding data content 1 to the log file.

[0200] S603. In response to the write request for the first object, change the version number of the first object to v2.

[0201] v2 > v1.

[0202] The write request can include a data modification request, a data deletion request, or a data insertion request.

[0203] S604. Write the version number v2 of the first object and the corresponding data content 2 to the log file.

[0204] In this application, data content 2 may include an operation type for the first object, and may also include data content corresponding to the operation type.

[0205] S603 and S604 can be executed repeatedly. Each time they are executed, the version number of the first object will be incremented, such as from v2 to v3, from v3 to v4, etc. The data content corresponding to different version numbers is also basically different.

[0206] The process of taking a snapshot of the first object in memory includes:

[0207] The first version number and first data content of the first object written in the snapshot file are related to the snapshot time and may include:

[0208] S611. If the snapshot time for the first object is before S601, then the snapshot file does not contain the first object.

[0209] S612. If the snapshot time of the first object is after S601 and before S603, then the version number v1 of the first object and the data content 1 corresponding to v1 are written to the snapshot file at once.

[0210] S613. If the snapshot time of the first object is after S603, then the version number v2 of the first object and the data content 2 corresponding to v2 are written to the snapshot file at once.

[0211] Of course, if the snapshot time is during the repeated execution process, the version number of the first object during the repeated execution process, as well as the corresponding data content, will be written to the snapshot file all at once.

[0212] The steps for restoring data corresponding to memory include:

[0213] In this embodiment, taking the version number of the first object in memory at the time of memory crash as v2 and the data content as data content 2 as an example, the memory recovery process is related to the snapshot processing process and may include:

[0214] S621. Restore the first version number and first data content of the first object in the snapshot file to memory.

[0215] Then, different steps are executed according to different situations in the snapshot file. If the situation in the snapshot file is S611, then S622 is executed; if the situation in the snapshot file is S612, then S624 is executed; if the situation in the snapshot file is S613, then S626 is executed.

[0216] S622. If the situation in the snapshot file is S611, restore the version number v1 and data content 1 of the first object in the log file to memory.

[0217] S623. Restore the version number v2 and data content 2 of the first object in the log file to memory, and delete the version number v1 and data content 1.

[0218] S624. If the situation in the snapshot file is S612, compare the version number v1 of the first object in the log file with the version number v1 of the first object that already exists in memory. If the version numbers are the same, discard the version number v1 and data content 1 of the first object recovered from the log file.

[0219] S625. Compare the version number v2 of the first object in the log file with the version number v1 of the first object that already exists in memory, determine that v2 > v1, apply data content 2 to data content 1 to obtain the restored data content, and mark the restored data content with v2.

[0220] S626. If the situation in the snapshot file is S613, compare the version number v1 of the first object in the log file with the version number v2 of the first object that already exists in memory, determine that v1 < v2, and discard the version number v1 and data content 1 of the first object recovered from the log file.

[0221] S627. Compare the version number v2 of the first object in the log file with the version number v2 of the first object that already exists in memory, determine that v1 = v2, and discard the version number v2 and data content 2 of the first object recovered from the log file.

[0222] In summary, during the snapshot process: for at least one first object whose length is less than a first threshold, the first version number and first data content of the first object are written to the snapshot file at once; in this way, for small objects, less processor resources and input / output (I / O) resources are used to quickly complete the one-time writing to the snapshot file, which can achieve fast writing of small objects without affecting other database operations.

[0223] In the memory recovery process: if the length of the first object is less than the first threshold, the first version number and the first data content of the first object in the snapshot file are recovered into memory; if the second version number of the first object in the log file is greater than the first version number, the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number; if the second version number of the first object in the log file is not greater than the first version number, the second version number and the second data content of the first object recovered from the log file are discarded.

[0224] In this application, the second data content can be an operation on the first object, or an operation plus data content.

[0225] Applying the second data content to the first data content to obtain the restored data content can be divided into several scenarios: First, if the operation corresponding to the second data content is an insertion operation, then the first data content is merged with the data content corresponding to the insertion operation; second, if the operation corresponding to the second data content is a modification operation, then the data content corresponding to the modification operation is used to replace the corresponding data in the first data content; third, if the operation corresponding to the second data content is a deletion operation, then the data content corresponding to the deletion operation is deleted from the first data content; fourth, if the operation corresponding to the second data content is a copy operation, then the data content corresponding to the second version number is the same as the first data content, and either the first data content or the data content corresponding to the copy operation is deleted.

[0226] For more information on the above steps, please refer to [link / reference]. Figure 6B Use the example diagram shown to understand. Figure 6BAs shown, in memory, at time point 1, the initial version number and corresponding data content of object 1 (object1, Obj1) were written, respectively v=1 and the corresponding data content = 1, which can be represented as Obj1, 1 (v=1); and the initial version number and corresponding data content of Obj2 were also written, respectively v=1 and the corresponding data content = 25, which can be represented as Obj2, 25 (v=1). At time point 2, both Obj1 and Obj2 were updated. The version number of Obj1 changed to v=2, and the corresponding data content = 2, which can be represented as Obj1, 2 (v=2); the version number of Obj2 changed to v=2, and the corresponding data content = 28, which can be represented as Obj2, 28 (v=2). At time points 1 and 2, the data generated from processing write requests, including Obj1,1 (v=1); Obj2,25 (v=1); Obj1,2 (v=2); and Obj2,28 (v=2), will be written to the log file. Figure 6B Although not explicitly shown, the corresponding process will still be executed. Of course, as time goes on, after data content of higher versions such as Obj1 and Obj2 is written, the lower version data content in the log file can be deleted, or it can be retained; this application does not limit this.

[0227] The snapshot period begins at time point 3 and extends from time point 3 to time point 5. During this snapshot period, memory contains Obj1 and Obj2. At time point 3, a write request for Obj1 is processed, changing its version number to v=3, and the corresponding data content to 3, which can be represented as Obj1,3(v=3). Therefore, within the snapshot period, memory contains Obj1,3(v=3) and Obj2,28(v=2). The content written to the snapshot file is Obj1,3(v=3) and Obj2,28(v=2).

[0228] After the snapshot period ended, at time point 6, a write request for Obj2 was processed. The version number of Obj2 changed to v=3, and the corresponding data content was 29, which can be represented as Obj2,29(v=3). Simultaneously, Obj2,29(v=3) was written to the log file. At time point 7, a write request for Obj1 was processed. The version number of Obj1 changed to v=4, and the corresponding data content was 4, which can be represented as Obj1,4(v=4). Simultaneously, Obj1,4(v=4) was written to the log file. Additionally, at time point 7, a write request for Obj2 was also processed. The version number of Obj2 changed to v=4, and the corresponding data content was 30, which can be represented as Obj2,30(v=4). Simultaneously, Obj2,30(v=4) was written to the log file.

[0229] If a memory crash occurs at time point 8, the objects in memory before the crash are Obj1,4 (v=4) and Obj2,30 (v=4).

[0230] The memory recovery process can be as follows: First, write Obj1,3 (v=3) and Obj2,28 (v=2) from the snapshot file into memory. Then, using the log file, first recover Obj1,3 (v=3) from the log file. By comparing the version number of the recovered Obj1 in memory, if the version numbers are the same, delete the Obj1,3 (v=3) recovered from the log file. Next, recover Obj2,29 (v=3) from the log file. It is determined that the version number v=3 of Obj2 recovered from the log file is greater than v=2 recovered from the snapshot file. Therefore, Obj2,28 (v=2) recovered from the snapshot file is deleted from memory, and Obj2,29 (v=3) is retained. Then, recover Obj1,4 (v=4) from the log file. By comparing the version number of Obj1, it is determined that v=4 is greater than the original v=3 in memory. Therefore, Obj1,3 (v=3) is deleted, and Obj1,4 (v=4) is retained. Similarly, recover Obj2,30 (v=4) from the log file. By comparing the version number of Obj2, it is determined that v=4 is greater than the original v=3 in memory. Therefore, Obj2,29 (v=3) is deleted, and Obj2,30 (v=4) is retained. This completes the memory recovery, and the data is consistent with the data in memory at the time of the crash at time 8.

[0231] It's important to note that the content in the log file will differ depending on the time of the memory crash. However, the general idea is to first use the snapshot file for memory recovery, then use the log file. If the version number of an object in the log file is less than or equal to (but not greater than) the version number of the corresponding object in the snapshot file, the data in the snapshot file is used, and the data recovered from the log file is discarded. If the version number of an object in the log file is greater than the version number of the corresponding object in the snapshot file, the log file is applied to the snapshot file to obtain the recovered data.

[0232] In the solution provided in this application embodiment, for small objects, the first version number and first data content of the first object in the snapshot file are first restored to memory. Then, the first version number restored in memory is determined by comparing the first version number and the second version number. If the second version number is greater than the first version number, the second data content needs to be applied to the first data content. This ensures that the data restored to memory is consistent with the data before the memory crash. If the second version number is not greater than the first version number, it means that the first version number is already a larger version number, and the second version number and second data content of the first object restored from the log file can be discarded.

[0233] in addition, Figure 6AIn the example, the operation corresponding to v2 is an update; if the operation corresponding to v2 is a delete operation, that is, first writing the version number v1 and data content 1 of the first object, and then deleting the first object, then... Figure 6A The snapshot files in different branches are as follows:

[0234] S611 scenario: The snapshot file does not contain the first object.

[0235] S612 scenario: The snapshot file contains the version number v1 and data content 1 of the first object.

[0236] S613 scenario: The snapshot file does not contain the first object because the snapshot task was unable to scan the first object after it was deleted.

[0237] like Figure 7A As shown, the data processing procedure for the first object of the second type includes:

[0238] The steps for writing the first object into memory and writing it to the log file include:

[0239] S701. Write the first object into memory for the first time and assign an initial version number v1 to the first object.

[0240] S702. Write the version number v1 of the first object and the corresponding data content 1 to the log file.

[0241] S703. In response to the write request for the first object, change the version number of the first object to v2.

[0242] v2 > v1.

[0243] The write request can include a data modification request or a data deletion request.

[0244] S704. Write the version number v2 of the first object and the corresponding data content 2 to the log file.

[0245] S703 and S704 can be executed repeatedly. Each time they are executed, the version number of the first object will be incremented, such as from v2 to v3, from v3 to v4, etc. The data content corresponding to different version numbers is also basically different.

[0246] The process of taking a snapshot of the first object in memory includes:

[0247] The first version number and first data content of the first object written in the snapshot file are related to the snapshot time and may include:

[0248] S711. If the snapshot time for the first object is before S701, then the snapshot file does not contain the first object.

[0249] S712. If the snapshot time of the first object is after S701 and before S703, then the version number v1 of the first object is written to the snapshot file all at once, and the data content 1 is written to the snapshot file multiple times.

[0250] Data content 1 may include multiple sub-objects, and at least one sub-object may be written each time. Each sub-object has at least one identifier, which is used to recover the data content corresponding to the at least one sub-object. In this application, each sub-object may have a single identifier, or multiple sub-objects may share a single identifier. One or more sub-objects with the same identifier are typically written to the snapshot file in a single operation.

[0251] For the process of writing the first object of the second type to the snapshot file, please refer to [link / reference]. Figure 7B To understand. For example Figure 7B As shown, taking Obj3 as an example, during the snapshot period, the version number of Obj3 is v=1. Obj3 includes 5 sub-objects, namely subobj1, subobj2, subobj3, subobj4, and subobj5. The identifiers of these 5 sub-objects are built into Obj3.

[0252] The snapshot process for Obj3 can be performed by first writing the version number v=1 of Obj3 and subobj1 to the snapshot file, which is the first write. After the first write, subobj2 and subobj3 are written to the snapshot file, which is the second write. After the second write, subobj4 and subobj5 are written to the snapshot file, which is the third write.

[0253] If a write request for Obj3 is processed during the process of writing to the snapshot file in stages, then some or all of the sub-objects of Obj3 can be written to the log file. For example, if the received write request in the third write operation indicates modification of subobj1 of Obj3, then Obj3, v=2, subobj1 can be written to the log file. Alternatively, Obj3, v=2, subobj1, subobj2, subobj3, subobj4, and subobj5 can all be written to the log file, but subobj2, subobj3, subobj4, and subobj5 will retain their original data content.

[0254] S713. If the snapshot time of the first object is after S703, then the version number v2 of the first object is written to the snapshot file all at once, and the data content 2 is written to the snapshot file in multiple parts.

[0255] The sub-objects included in data content 2 may be the same as or different from the sub-objects included in data content 1. During the writing of some sub-objects of the first object, other sub-objects may be modified or deleted.

[0256] Of course, if the snapshot time is during the repeated execution process, the version number of the first object during the repeated execution process will be written to the snapshot file all at once, and the corresponding data content will be written to the snapshot file multiple times.

[0257] The process of writing data content in S712 or S713 can be as follows: first, write a portion of data content 1 or data content 2, and then determine whether data content 1 or data content 2 has been completely written. If not, continue writing until data content 1 or data content 2 has been completely written to the snapshot file.

[0258] The steps for restoring data corresponding to memory include:

[0259] In this embodiment, taking the version number of the first object in memory at the time of memory crash as v2 and the data content as data content 2 as an example, the memory recovery process is related to the snapshot processing process and may include:

[0260] S721. Restore the first version number and first data content of the first object in the snapshot file to memory.

[0261] Then, different steps are executed according to different situations in the snapshot file. If the situation in the snapshot file is S711, then S722 is executed; if the situation in the snapshot file is S712, then S724 is executed; if the situation in the snapshot file is S713, then S726 is executed.

[0262] S722. If the situation in the snapshot file is S711, restore the version number v1 and data content 1 of the first object in the log file to memory.

[0263] S723. Restore the version number v2 and data content 2 of the first object in the log file to memory, and delete the version number v1 and data content 1.

[0264] S724. If the situation in the snapshot file is S712, compare the version number v1 of the first object in the log file with the version number v1 of the first object that already exists in memory. If the version numbers are the same, discard the version number v1 and data content 1 of the first object recovered from the log file.

[0265] S725. Compare the version number v2 of the first object in the log file with the version number v1 of the first object that already exists in memory, determine that v2 > v1, apply data content 2 to data content 1 to obtain the restored data content, and mark the restored data content with v2.

[0266] S726. If the situation in the snapshot file is S713, compare the version number v1 of the first object in the log file with the version number v2 of the first object that already exists in memory, determine that v1 < v2, and discard the version number v1 and data content 1 of the first object recovered from the log file.

[0267] S727. Compare the version number v2 of the first object in the log file with the version number v2 of the first object that already exists in memory, determine that v1 = v2, and discard the version number v2 and data content 2 of the first object recovered from the log file.

[0268] In summary, during the snapshot process: if the length of the first object is greater than the first threshold and the first data content of the first object is not related to each other, i.e., the first object of the second type, the first version number of the first object is written to the snapshot file once, and the first data content of the first object is written to the snapshot file multiple times.

[0269] In the memory recovery phase: if the length of the first object is greater than the first threshold, and the first data content of the first object is not related to each other, the data corresponding to the memory is recovered, including: recovering the first version number and the first data content of the first object in the snapshot file into memory; if the second version number of the first object in the log file is greater than the first version number, the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number; if the second version number of the first object in the log file is not greater than the first version number, the second version number and the second data content of the first object recovered from the log file are discarded.

[0270] In this application, "not related to each other" means that the data content of different sub-objects of the first object does not need to maintain a sequential or relative relationship. This type of data is the reentrant data described above.

[0271] In this embodiment of the application, for reentrant data, the data in memory can be recovered by comparing the version number. This ensures that the data recovered to memory is consistent with the data before the memory crash.

[0272] The memory recovery process for large objects of the second type in this application is basically the same as that for small objects of the first type. Memory recovery only requires comparing the version number. Please refer to the above. Figure 6B To understand this, we need to look at examples.

[0273] like Figure 8A As shown, the data processing procedure for the first object of the third type includes:

[0274] The steps for writing the first object into memory and writing it to the log file include:

[0275] S801. Write the first object into memory for the first time, assign an initial version number v1 to the first object, and assign identifiers to the child objects of the first object.

[0276] In this application, a unique identifier can be assigned to each sub-object of the first object, or a unique identifier can be assigned to multiple sub-objects; this application does not limit this.

[0277] like Figure 8B As shown, taking Obj3 as an example, during the snapshot period, the version number v of Obj3 is 1. Obj3 includes 5 sub-objects. When Obj3 is written to memory, identifiers are assigned to these 5 sub-objects, such as... Figure 8B In this application, the assigned identifiers are subobj1, subobj2, subobj3, subobj4, and subobj5. Typically, identifiers are assigned to sub-objects in sequence.

[0278] S802. Write the version number v1 of the first object, the identifier of the sub-object, and the corresponding data content 1 to the log file.

[0279] S803. In response to a write request for the first object, change the version number of the first object to v2. If the write request adds a target sub-object to the first object, assign an identifier to the new target sub-object.

[0280] v2 > v1.

[0281] The write request can also be a data modification request or a data deletion request.

[0282] If no new child objects are added, there is no need to assign an identifier.

[0283] S804. Write the version number v2 of the first object, the identifier of the target sub-object, the corresponding data content 2, the identifier of the predecessor sub-object of the target sub-object, and the identifier of the successor sub-object associated with the identifier of the target sub-object to the log file.

[0284] In this application, the predecessor sub-object, target sub-object, and successor sub-object are sub-objects arranged sequentially. The predecessor sub-object is the neighbor preceding the target sub-object, and the successor sub-object is the neighbor following the target sub-object. The relationship between these three can be understood by referring to 8C, such as... Figure 8C As shown, if the write request is an insertion operation on the target sub-object, that is, inserting the target sub-object, then... Figure 8CIn the scenario shown, the insertion operation instructs the insertion of a target sub-object into subobj3 and subobj4, and assigns an identifier, subobj6, to this target sub-object. Therefore, subobj3 is the predecessor sub-object of subobj6, and subobj4 is the successor sub-object of subobj6. The version number v2, the identifiers of the three sub-objects subobj3, subobj6, and subobj4, and the data content of subobj6 are then written to the log file.

[0285] If the write request is a modification or deletion operation on the target sub-object, that is, modifying or deleting the target sub-object. For example... Figure 8D In the scenario shown, if the modification operation instructs to modify subobj3, or the deletion operation instructs to delete subobj3, then subobj2 is the predecessor child object of subobj3, and subobj4 is the successor child object of subobj3. The identifiers of subobj2, subobj3, and subobj4 satisfy the order relationship. If it is a modification operation, the version number v2, the identifiers of the three child objects subobj2, subobj3, and subobj4, and the modified data content of subobj3 are written to the log file. If it is a deletion operation, the version number v2, the identifiers of the three child objects subobj2, subobj3, and subobj4 are written to the log file.

[0286] S803 and S804 can be executed repeatedly. Each time they are executed, the version number of the first object will be incremented, such as from v2 to v3, from v3 to v4, etc. The data content corresponding to different version numbers is also basically different.

[0287] The process of taking a snapshot of the first object in memory includes:

[0288] The first version number and first data content of the first object written in the snapshot file are related to the snapshot time and may include:

[0289] S811. If the snapshot time for the first object is before S801, then the snapshot file does not contain the first object.

[0290] S812. If the snapshot time of the first object is after S801 and before S803, then the version number v1 of the first object is written to the snapshot file all at once, and the data content 1 is written to the snapshot file multiple times.

[0291] Data content 1 may include multiple sub-objects, and at least one sub-object may be written each time. Each sub-object has at least one identifier, which is used to recover the data content corresponding to the at least one sub-object. In this application, each sub-object may have a single identifier, or multiple sub-objects may share a single identifier. One or more sub-objects with the same identifier are typically written to the snapshot file in a single operation.

[0292] For the process of writing the first object of the third type to the snapshot file, please refer to [link / reference]. Figure 7B The introduction has already been explained in detail, so I will not repeat it here.

[0293] S813. If the snapshot time of the first object is after S803, then the version number v2 of the first object is written to the snapshot file all at once, and the data content 2 is written to the snapshot file in multiple parts.

[0294] The sub-objects included in data content 2 may be the same as or different from the sub-objects included in data content 1. During the writing of some sub-objects of the first object, other sub-objects may be modified or deleted.

[0295] Of course, if the snapshot time is during the repeated execution process, the version number of the first object during the repeated execution process will be written to the snapshot file all at once, and the corresponding data content will be written to the snapshot file multiple times.

[0296] The process of writing data content in multiple steps in S812 or S813 can be as follows: first, write a portion of data content 1 or data content 2, and then determine whether data content 1 or data content 2 has been completely written. If not, continue writing until data content 1 or data content 2 has been completely written to the snapshot file.

[0297] The steps for restoring data corresponding to memory include:

[0298] In this embodiment, taking the version number of the first object in memory at the time of memory crash as v2 and the data content as data content 2 as an example, the memory recovery process is related to the snapshot processing process and may include:

[0299] S821. Restore the first version number and first data content of the first object in the snapshot file to memory.

[0300] Then, different steps are executed according to different situations in the snapshot file. If the situation in the snapshot file is S811, then S822 is executed; if the situation in the snapshot file is S812, then S824 is executed; if the situation in the snapshot file is S813, then S828 is executed.

[0301] S822. If the situation in the snapshot file is S811, restore the version number v1 and data content 1 of the first object in the log file to memory.

[0302] S823. Restore the version number v2 and data content 2 of the first object in the log file to memory, and delete the version number v1 and data content 1.

[0303] S824. If the situation in the snapshot file is S812, compare the version number v1 of the first object in the log file with the version number v1 of the first object that already exists in memory. If the version numbers are the same, discard the version number v1 and data content 1 of the first object recovered from the log file.

[0304] S825. Compare the version number v2 of the first object in the log file with the version number v1 of the first object that already exists in memory, and determine that v2 > v1.

[0305] S826. Determine whether the verification rules are met; if the verification rules are met, execute S827; if the verification rules are not met, discard the version number v2 and the corresponding data content 2 of the first object restored to memory.

[0306] In this application, the verification rules include at least one of the following:

[0307] If the predecessor child object of the target child object in the log file is used to find the predecessor child object in memory, and the successor child object is found using the successor child object, both will succeed.

[0308] If the target sub-object in the log file corresponds to the insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, and both the predecessor sub-object and the successor sub-object are adjacent to the target sub-object;

[0309] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object must satisfy an order relationship.

[0310] In this application, at least one of the following validation rules is not met:

[0311] If using the identifier of the predecessor sub-object of the target sub-object in the log file to find the predecessor sub-object in memory fails, or using the identifier of the successor sub-object to find the successor sub-object fails;

[0312] If the target sub-object in the log file corresponds to an insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, but the predecessor sub-object and the successor sub-object are not adjacent to the target sub-object;

[0313] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object do not satisfy the order relationship.

[0314] S827. Apply data content 2 to data content 1 to obtain the restored data content, and mark the restored data content with v2.

[0315] S826. If the situation in the snapshot file is S813, compare the version number v1 of the first object in the log file with the version number v2 of the first object that already exists in memory, determine that v1 < v2, and discard the version number v1 and data content 1 of the first object recovered from the log file.

[0316] S828. Compare the version number v2 of the first object in the log file with the version number v2 of the first object that already exists in memory, determine that v1 = v2, and discard the version number v2 and data content 2 of the first object recovered from the log file.

[0317] In summary, for the third type of large objects, during the memory writing process, when writing the first object to memory, at least one identifier is assigned to each of the first object's child objects; each child object corresponds to one identifier. This way, by assigning unique identifiers to child objects that do not have identifiers, it facilitates subsequent writing of the first object to snapshot files or log files and also ensures data consistency after recovery.

[0318] In the snapshot phase: if the length of the first object is greater than the first threshold and the first data content of the first object is related to each other, that is, the first object of the third type, the first version number and the first data content of the first object in the snapshot file are restored to memory; according to the first version number and the second version number of the first object, and the verification rules, the second version number and the second data content of the first object restored from the log file are used or discarded.

[0319] In this application, "interrelated" refers to the sequential or relative relationship between the data contents of different sub-objects of the first object. This type of data is the non-reentrant data described above.

[0320] In the memory recovery phase: If the second version number of the first object is not greater than the first version number, or if the first object does not meet the verification rules, then the second version number and the second data content of the first object recovered from the log file are discarded. If the second version number of the first object is greater than the first version number, and the first object meets the verification rules, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number.

[0321] In the above embodiments, a reasonable first threshold can be set according to the actual business scenario. When the length of the first object is less than the first threshold, the entire object can be persisted at once in the snapshot task, avoiding multiple persistence steps. That is, for the first type of first object, the execution process of allocating a unique identifier (id) can be avoided. When the object length exceeds the first threshold, a unique id is then assigned to all child objects of that object, and the unique id information is persisted. In addition, in this embodiment, a single unique id can be assigned to multiple child objects, which can reduce the overhead of allocating and storing child object ids. Furthermore, in this embodiment, a suitable memory data structure can be introduced, such as using a set / hash structure, to accelerate the lookup speed of the unique id.

[0322] The data processing method described above in this application embodiment can be applied to multiple scenarios, which will be described separately below.

[0323] like Figure 9A As shown, the data processing in a single-machine production environment can include: Regular data backups are performed in the production environment. At a certain point, a full object scan of the database is initiated and written to a snapshot file. Normal business read / write operations are not blocked; new data modification commands are simultaneously written asynchronously to the log file. Once the snapshot file is generated, combined with the log file at that moment, a complete database snapshot at that point is formed. During data recovery, the production environment first restores the snapshot file. After the snapshot file completes data recovery, log file recovery is initiated. The database will recover or discard logs based on the version number in the object header. Finally, the database is restored to a snapshot at a point before the backup, ensuring data consistency.

[0324] Figure 9A The scenario shown does not distinguish between local disks and shared storage. Applying the data processing method described in this application, asynchronous data replication can be performed without blocking business operations, generating persistent files composed of logs and snapshots. This supports backups from the production environment to local or off-site environments, while ensuring seamless business operation and uninterrupted service. When the system suffers unexpected events such as natural disasters, human error, or system failures, the production environment can recover data using the backed-up logs and snapshot persistent files, restoring data to its pre-failure state and ensuring data consistency before and after the failure.

[0325] like Figure 9BAs shown, in the distributed cluster system, there are three master nodes. Master node 1 assigns partitions 1 and 4, master node 2 assigns partitions 2 and 5, and master node 3 assigns partitions 3 and 6. Their persistent files are all stored in shared storage. Black squares represent snapshot files, and white squares represent log files. This application enables partition data migration in this deployment mode. When one master node fails, its assigned partitions can be reassigned to other master nodes. The other master nodes can then recover data by retrieving the logs and snapshot persistent files, thus completing the partition migration between nodes.

[0326] like Figure 9B In the scenario shown, each master node in the distributed cluster system can record log files in real time and periodically generate snapshot files. Object version numbers are retained in the objects recorded in both the logs and snapshot files. If, at some point, the system's management service detects a failure or crash of master node 1, the management service reassigns partitions 1 and 4, originally belonging to master node 1, to nodes 2 and 3, respectively. Nodes 2 and 3 read the log and snapshot files of partitions 1 and 4 from shared storage. Nodes 2 and 3 first restore the snapshot files, then restore the log files, and use the object version numbers to either restore or discard the logs, ultimately completing the partition migration and ensuring data consistency with the data before node 1's failure.

[0327] In addition to the partition migration caused by failure or crash, the above process can also be used if a distributed system based on shared storage needs to be expanded or reduced in size.

[0328] like Figure 10A As shown, in a shared storage application scenario, the solution provided in this application can be used to expand to a new standby node. The standby node restores data from the log and snapshot files in the shared storage, achieving data replication from the primary node to the new standby node and ensuring eventual data consistency with the primary node. The process may include: the primary node recording log files in real time and periodically generating snapshot files. When system expansion requires a new standby node, the new standby node first restores the snapshot files from the shared storage, then restores the log files, and restores or discards the logs according to their version numbers. This completes the data replication from the primary node to the new standby node.

[0329] like Figure 10BAs shown, in local disk application scenarios, the difference from shared storage is that the snapshot file of the new standby node is sent by the master node, and the new standby node receives message forwarding from the master node in real time, no longer restoring log files recorded in storage. The process can include: the master node records log files in real time and periodically generates snapshot files. When system expansion requires a new standby node, the master node sends the snapshot file to the standby node. The standby node restores the data in the snapshot file and responds to message forwarding from the master node. Based on the version number of the restored data objects, it restores or discards the data objects to ensure data consistency with the master node. In a distributed cluster system based on local disks, node expansion or contraction can also be performed using the above method. Figure 10A The process is shown.

[0330] The above embodiments describe data processing methods. The data processing apparatus 1100 provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0331] like Figure 11 As shown, the data processing device 1100 provided in this application embodiment can be the data node described above, and the data processing device 1100 includes:

[0332] The first processing unit 1101 is configured to write at least one first object in memory into a snapshot file during a first time period; wherein the snapshot file includes a first version number and first data content for each first object;

[0333] The second processing unit 1102 is used to write the second version number and second data content of the second object corresponding to the write request within the first time period into a log file; wherein the second object is contained in at least one first object;

[0334] The snapshot file and log file are used to restore the data corresponding to the first object.

[0335] Optionally, the first processing unit 1101 is specifically used for:

[0336] For at least one first object whose length is less than a first threshold, the first version number and first data content of the first object are written to the snapshot file at once;

[0337] For at least one first object whose length is greater than a first threshold, the first version number of the first object is written to the snapshot file once, and the first data content of the first object is written to the snapshot file multiple times.

[0338] Optionally, the first processing unit 1101 is specifically used for: if the length of the first object is greater than the first threshold, the first object includes multiple sub-objects; writing the multiple sub-objects into the snapshot file in multiple batches; wherein, at least one sub-object is written each time, and at least one sub-object has at least one identifier, and at least one identifier is used to recover the data content corresponding to at least one sub-object.

[0339] Optionally, the first processing unit 1101 is further configured to: if the sub-objects of the first object do not have at least one identifier, when writing the first object to memory, allocate at least one identifier to the sub-objects of the first object; wherein each sub-object corresponds to one identifier.

[0340] Optionally, the second processing unit 1102 is further configured to, if the write request is an insert request, modification request, or deletion request for the target sub-object, associate the identifiers of the predecessor sub-objects adjacent to the target sub-object and the identifiers of the successor sub-objects with the identifier of the target sub-object and write them to a log file.

[0341] Optionally, the data processing apparatus 1100 further includes a third processing unit 1103;

[0342] The third processing unit 1103 is configured to: restore the first version number and first data content of the first object in the snapshot file to memory if the length of the first object is less than the first threshold; if the second version number of the first object in the log file is greater than the first version number, apply the second data content to the first data content to obtain the restored data content, and mark the restored data content with the second version number; if the second version number of the first object in the log file is not greater than the first version number, discard the second version number and the second data content of the first object restored from the log file.

[0343] Optionally, the third processing unit 1103 is configured to: restore the first version number and first data content of the first object in the snapshot file to memory if the length of the first object is greater than the first threshold and the first data content of the first object is not related to each other; if the second version number of the first object in the log file is greater than the first version number, apply the second data content to the first data content to obtain the restored data content, and mark the restored data content with the second version number; if the second version number of the first object in the log file is not greater than the first version number, discard the second version number and the second data content of the first object restored from the log file.

[0344] Optionally, the third processing unit 1103 is configured to: if the length of the first object is greater than a first threshold and the first data contents of the first objects are related to each other, restore the first version number and the first data contents of the first object in the snapshot file to memory; and use or discard the log file according to the first version number and the second version number of the first object and the verification rules.

[0345] Optionally, the third processing unit 1103 is configured to discard the second version number and the second data content of the first object recovered from the log file if the second version number of the first object is not greater than the first version number, or if the first object does not meet the verification rules.

[0346] Optionally, the third processing unit 1103 is configured to apply the second data content to the first data content if the second version number of the first object is greater than the first version number and the first object meets the verification rules, thereby obtaining the restored data content, and marking the restored data content with the second version number.

[0347] Optionally, the validation rules include at least one of the following:

[0348] If the predecessor child object of the target child object in the log file is used to find the predecessor child object in memory, and the successor child object is found using the successor child object, both will succeed.

[0349] If the target sub-object in the log file corresponds to the insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, and both the predecessor sub-object and the successor sub-object are adjacent to the target sub-object;

[0350] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object must satisfy an order relationship.

[0351] Optionally, failing to meet the validation rules includes at least one of the following:

[0352] If using the identifier of the predecessor sub-object of the target sub-object in the log file to find the predecessor sub-object in memory fails, or using the identifier of the successor sub-object to find the successor sub-object fails;

[0353] If the target sub-object in the log file corresponds to an insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, but the predecessor sub-object and the successor sub-object are not adjacent to the target sub-object;

[0354] If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object do not satisfy the order relationship.

[0355] The above-mentioned contents of the first processing unit 1101, the second processing unit 1103 and the third processing unit 1103 of the data processing device 1100 can be understood by referring to the relevant contents of the data nodes in the aforementioned method embodiment section, and will not be repeated here.

[0356] Figure 12The diagram shown illustrates a possible logical structure of the data processing device 1200 described in the embodiments of this application. The data processing device 1200 includes a processor 1201, a communication interface 1202, a memory 1203, and a bus 1204. The processor 1201, communication interface 1202, and memory 1203 are interconnected via the bus 1204. In the embodiments of this application, the processor 1201 is used to control and manage the operation of the data processing device 1200; for example, the processor 1201 is used to execute... Figures 4 to 8D The process includes steps related to processing, such as S401, S402, etc. The communication interface 1202 supports communication between the data processing device 1200 and other components. For example, the communication interface 1202 can execute steps related to receiving or sending data in the above method embodiments, such as receiving read requests or write requests. The memory 1203 stores the program code and data of the database system 120.

[0357] The processor 1201 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 1204 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0358] See Figure 13 This application also provides a distributed database system, including: a hardware layer 1307 and a virtual machine monitor (VMM) 1301 running on the hardware layer 1307, and multiple virtual machines 1302. A virtual machine can serve as a read node or a write node in the distributed database system. Optionally, a virtual machine can also be designated as a coordinating node.

[0359] Specifically, Virtual Machine 1302 is a virtual computer simulated on public hardware resources using virtual machine software. Operating systems and applications can be installed on the virtual machine, and it can also access network resources. For applications running in the virtual machine, it's as if they are working on a real computer.

[0360] Hardware Layer 1307: The hardware platform running in the virtualized environment, which can be abstracted from the hardware resources of one or more physical hosts. The hardware layer may include various hardware components, such as a processor 1304 (e.g., CPU) and memory 1305, as well as a network interface card 1303 (e.g., RDMA network card), high-speed / low-speed input / output (I / O) devices, and other devices with specific processing functions.

[0361] Virtual machine 1302, based on the VMM and the hardware resources provided by hardware layer 1307, runs executable programs to achieve the above. Figures 4 to 10B In the relevant embodiments, some or all of the functions of the coordination node are described. For the sake of brevity, these will not be elaborated further here.

[0362] Furthermore, the distributed database system may also include a host: acting as a management layer to manage and allocate hardware resources; presenting a virtual hardware platform for virtual machines; and implementing virtual machine scheduling and isolation. The host may be a Virtual Machine Monitor (VMM); or it may be a combination of a VMM and a privileged virtual machine. The virtual hardware platform provides various hardware resources to the virtual machines running on it, such as virtual processors (e.g., VCPUs), virtual memory, virtual disks, virtual network cards, etc. The virtual disk may correspond to a file or a logical block device on the host. Virtual machines run on the virtual hardware platform prepared for them by the host, and one or more virtual machines run on the host. The VCPU of virtual machine 1302 executes an executable program stored in its corresponding virtual memory to implement or perform the method steps described in the above-described method embodiments of the present invention. For example, to implement the above... Figures 4 to 10B In related embodiments, some or all of the functions of the coordination node are used.

[0363] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When at least one processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figures 4 to 10B The data processing method described in some embodiments.

[0364] In another embodiment of this application, a computer program product is also provided, comprising computer-executable instructions stored in a computer-readable storage medium; at least one processor of the device can read the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to cause the device to perform the above-described actions. Figures 4 to 10B The data processing method described in some embodiments.

[0365] In another embodiment of this application, a chip system is also provided, the chip system including a processor for supporting the coordination node to implement the above. Figures 4 to 10B The data processing method described in some embodiments. In one possible design, the chip system may further include a memory for coordinating necessary program instructions and data for the nodes. The chip system may be composed of chips or may include chips and other discrete devices.

[0366] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0367] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0368] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0369] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0370] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0371] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data processing method, characterized in that, include: In a first time period, at least one first object in memory is written to a snapshot file; wherein the snapshot file includes a first version number and first data content for each first object; Write the second version number and second data content of the second object corresponding to the write request within the first time period to the log file; wherein, the second object is contained in the at least one first object; The first version number of the first object in the snapshot file and the second version number of the second object in the log file are used to restore the data corresponding to the memory.

2. The method according to claim 1, characterized in that, The step of writing at least one first object in memory to a snapshot file includes: For the first object whose length is less than the first threshold among the at least one first object, the first version number and the first data content of the first object are written to the snapshot file at once; For the first object whose length is greater than a first threshold among the at least one first object, the first version number of the first object is written to the snapshot file once, and the first data content of the first object is written to the snapshot file multiple times.

3. The method according to claim 2, characterized in that, If the length of the first object is greater than the first threshold, the first object includes multiple sub-objects; the first data content of the first object is written to the snapshot file in multiple parts, including: The multiple sub-objects are written to the snapshot file in multiple batches; wherein at least one sub-object is written each time, and the at least one sub-object has at least one identifier, and the at least one identifier is used to restore the data content corresponding to the at least one sub-object.

4. The method according to claim 3, characterized in that, If the child object of the first object does not itself possess the at least one identifier, the method further includes: When writing the first object to the memory, at least one identifier is assigned to the child objects of the first object; wherein each child object corresponds to one identifier.

5. The method according to claim 4, characterized in that, When writing the second version number and second data content of the second object corresponding to the write request within the first time period to the log file, the method further includes: If the write request is an insertion request, modification request, or deletion request for the target sub-object, the identifiers of the predecessor sub-objects adjacent to the target sub-object and the identifiers of the successor sub-objects are associated with the identifier of the target sub-object and written into the log file.

6. The method according to claim 2, characterized in that, If the length of the first object is less than the first threshold, the recovery of the data corresponding to the memory includes: Restore the first version number and the first data content of the first object in the snapshot file to the memory; If the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number. If the second version number of the first object in the log file is not greater than the first version number, then the second version number and the second data content of the first object recovered from the log file are discarded.

7. The method according to claim 3, characterized in that, If the length of the first object is greater than the first threshold, and the first data content of the first object is not related to each other, the recovery of the data corresponding to the memory includes: Restore the first version number and the first data content of the first object in the snapshot file to the memory; If the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number. If the second version number of the first object in the log file is not greater than the first version number, then the second version number and the second data content of the first object recovered from the log file are discarded.

8. The method according to claim 5, characterized in that, If the length of the first object is greater than the first threshold, and the first data contents of the first objects are correlated, the recovery of the data corresponding to the memory includes: Restore the first version number and the first data content of the first object in the snapshot file to the memory; Based on the first version number and the second version number of the first object, and the verification rules, the second version number and the second data content of the first object recovered from the log file are used or discarded.

9. The method according to claim 8, characterized in that, If the second version number of the first object is not greater than the first version number, or if the first object does not meet the verification rules, then the second version number and the second data content of the first object recovered from the log file are discarded.

10. The method according to claim 8, characterized in that, If the second version number of the first object is greater than the first version number, and the first object meets the verification rules, then the second data content is applied to the first data content to obtain the restored data content, and the restored data content is marked with the second version number.

11. The method according to claim 10, characterized in that, The verification rule includes at least one of the following: If the predecessor sub-object of the target sub-object in the log file is used to search for the predecessor sub-object in memory, and the successor sub-object is searched using the identifier of the successor sub-object, both searches are successful. If the target sub-object in the log file corresponds to an insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, and both the predecessor sub-object and the successor sub-object are adjacent to the target sub-object; If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object satisfy an order relationship.

12. The method according to claim 9, characterized in that, The failure to meet the verification rule includes at least one of the following: If using the identifier of the predecessor sub-object of the target sub-object in the log file to search for the predecessor sub-object in the memory fails, or using the identifier of the successor sub-object to search for the successor sub-object fails; If the target sub-object in the log file corresponds to an insertion request, both the identifier of the predecessor sub-object and the identifier of the successor sub-object exist, but the predecessor sub-object and the successor sub-object are not adjacent to the target sub-object; If the target sub-object in the log file corresponds to a modification or deletion operation, the identifiers of the predecessor sub-object, the target sub-object, and the successor sub-object do not satisfy the order relationship.

13. A data processing apparatus, characterized in that, include: A first processing unit is configured to write at least one first object in memory into a snapshot file during a first time period; wherein the snapshot file includes a first version number and first data content for each first object; The second processing unit is configured to write the second version number and second data content of the second object corresponding to the write request within the first time period into a log file; wherein the second object is included in the at least one first object; The snapshot file and the log file are used to restore the data corresponding to the first object.

14. The apparatus according to claim 13, characterized in that, The first processing unit is specifically used for: For the first object whose length is less than the first threshold among the at least one first object, the first version number and the first data content of the first object are written to the snapshot file at once; For the first object whose length is greater than a first threshold among the at least one first object, the first version number of the first object is written to the snapshot file once, and the first data content of the first object is written to the snapshot file multiple times.

15. The apparatus according to claim 14, characterized in that, The first processing unit is specifically used for: if the length of the first object is greater than the first threshold, the first object includes multiple sub-objects; writing the multiple sub-objects into the snapshot file in multiple parts; wherein, at least one sub-object is written each time, the at least one sub-object has at least one identifier, and the at least one identifier is used to restore the data content corresponding to the at least one sub-object.

16. The apparatus according to claim 15, characterized in that, The first processing unit is further configured to: if a sub-object of the first object does not have the at least one identifier, when writing the first object to the memory, allocate the at least one identifier to the sub-object of the first object; wherein each sub-object corresponds to one identifier.

17. The apparatus according to claim 16, characterized in that, The second processing unit is further configured to, if the write request is an insertion request, modification request, or deletion request for the target sub-object, associate the identifiers of the predecessor sub-objects adjacent to the target sub-object and the identifiers of the successor sub-objects with the identifier of the target sub-object and write them into the log file.

18. The apparatus according to claim 14, characterized in that, The device also includes a third processing unit. The third processing unit is configured to, if the length of the first object is less than the first threshold: Restore the first version number and the first data content of the first object in the snapshot file to the memory; If the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number. If the second version number of the first object in the log file is not greater than the first version number, then the second version number and the second data content of the first object recovered from the log file are discarded.

19. The apparatus according to claim 15, characterized in that, The device also includes a third processing unit. The third processing unit is configured to, if the length of the first object is greater than the first threshold, and the first data contents of the first objects are not related to each other: Restore the first version number and the first data content of the first object in the snapshot file to the memory; If the second version number of the first object in the log file is greater than the first version number, then the second data content is applied to the first data content to obtain the recovered data content, and the recovered data content is marked with the second version number. If the second version number of the first object in the log file is not greater than the first version number, then the second version number and the second data content of the first object recovered from the log file are discarded.

20. The apparatus according to claim 17, characterized in that, The device also includes a third processing unit. The third processing unit is configured to, if the length of the first object is greater than the first threshold, and the first data contents of the first objects are correlated: Restore the first version number and the first data content of the first object in the snapshot file to the memory; Based on the first version number and the second version number of the first object, and the verification rules, use the log file or discard the log file; If the second version number of the first object is not greater than the first version number, or if the first object does not meet the verification rules, then the second version number and the second data content of the first object recovered from the log file are discarded. If the second version number of the first object is greater than the first version number, and the first object meets the verification rules, then the second data content is applied to the first data content to obtain the restored data content, and the restored data content is marked with the second version number.

21. A data processing apparatus, characterized in that, Includes a processor and a computer-readable storage medium storing a computer program; The processor is coupled to the computer-readable storage medium, and the computer program, when executed by the processor, implements the method as described in any one of claims 1-12.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-12.

23. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer device, causes the computer device to perform the method as described in any one of claims 1-12.

24. A database system, characterized in that, It includes at least one data node, wherein the data node includes the data processing apparatus according to any one of claims 13-20.

Images