Data management method and device, electronic equipment and storage medium

Abstract

The present specification provides a data management method and device, an electronic device and a storage medium. The method comprises: obtaining an ID of a current task, wherein the ID of the current task is greater than the IDs of all started tasks for a shared storage layer; performing a data operation of the current task, and marking the ID of the current task for data uploaded to the shared storage layer in the data operation process; if the data operation of the current task is completed, generating a data record file and marking the ID of the current task for the data record file, and recording index information in the data record file, wherein the index information is used for indexing all data in the shared storage layer.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of database, and in particular, to a data management method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the exponential growth of data in the field of computer and the development of database technology, a shared storage mode is gradually born, that is, by setting a shared storage layer and a plurality of database nodes in communication with the shared storage layer, user data and the like can be stored in the shared storage layer for sharing by the plurality of database nodes, and the database nodes cache part of the data in the shared storage layer locally and provide data services to the outside according to the local cache.

[0003] The database nodes also need to upload data processing results, such as data writing results, data modification results, and the like, to the shared storage layer to update the data in the shared storage layer. However, in the related art, the database nodes under the shared storage architecture cannot guarantee the atomicity of data operations, thereby causing confusion or even errors in the data in the shared storage layer. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide a data management method and device, electronic equipment and storage medium.

[0005] To achieve the above object, one or more embodiments of the present specification provide the technical solutions as follows.

[0006] According to a first aspect of one or more embodiments of the present specification, a data management method is provided, and the method comprises:

[0007] obtaining an ID of a current task, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer;

[0008] performing a data operation of the current task, and marking the ID of the current task for data uploaded to the shared storage layer during the data operation;

[0009] if the data operation of the current task is completed, generating a data record file and marking the ID of the current task for the data record file, and recording index information in the data record file, wherein the index information is used to index all data in the shared storage layer.

[0010] In one possible embodiment of the present specification, the method further comprises:

[0011] generating a task record file and marking the ID of the current task for the task record file;

[0012] During the data operation of the current task, the operation execution progress recorded in the task record file and / or the data range of the data uploaded to the shared storage layer during the operation execution are recorded.

[0013] In one possible implementation of the present specification, if the data operation of the current task is completed, a data record file is generated, the ID of the current task is marked for the data record file, and index information is recorded in the data record file, including:

[0014] If the operation execution progress recorded in the task record file is completed, a data record file is generated, the ID of the current task is marked for the data record file, and index information is recorded in the data record file.

[0015] In one possible implementation of the present specification, the ID of the current task is obtained, including:

[0016] The ID record file with the largest ID is obtained, and the ID of the current task is obtained by incrementing the ID of the ID record file.

[0017] The ID record file corresponding to the ID of the current task is constructed.

[0018] In one possible implementation of the present specification, the ID record file with the largest ID is obtained, including:

[0019] The latest ID file is obtained, and the ID record file with the largest ID is obtained based on the ID in the latest ID file.

[0020] After obtaining the ID of the current task, the method further includes:

[0021] The ID in the latest ID file is updated.

[0022] In one possible implementation of the present specification, the data operation of the current task is performed, and the ID of the current task is marked for the data uploaded to the shared storage layer during the data operation, including:

[0023] The to-be-uploaded data of the current task is uploaded to the shared storage layer in the form of an object file.

[0024] For each object file in at least one object file uploaded to the shared storage layer, the ID of the current task and the upload order of the object file in the current task are used as the name of the object file.

[0025] In one possible implementation of the present specification, the to-be-uploaded data of the current task is uploaded to the shared storage layer in the form of an object file, including:

[0026] uploading data content in the to-be-uploaded data of the current task to the data content directory under the shared storage layer in the form of an object file, and uploading metadata in the to-be-uploaded data of the current task to the metadata directory under the shared storage layer in the form of an object file;

[0027] the object file is named by using the ID of the current task and the uploading sequence of the object file in the current task, including:

[0028] the object file is named by using the directory in which the object file is located, the ID of the current task, and the uploading sequence of the object file in the current task.

[0029] In one possible embodiment of the present specification, the method further includes:

[0030] if the lease is not expired, generating the task record file;

[0031] the operation execution progress is recorded in the task record file, including:

[0032] if the data operation of the current task is completed and the lease is not expired, recording the operation execution progress representing completion in the task record file;

[0033] if the data operation of the current task is completed, generating a data record file, marking the ID of the current task for the data record file, and recording index information in the data record file, including:

[0034] if the data operation of the current task is completed and the lease is not expired, generating a data record file, marking the ID of the current task for the data record file, and recording index information in the data record file;

[0035] of the plurality of data nodes in communication connection with the shared storage layer only one data node has the lease at the same time, and the data node having the lease is used to execute the current task.

[0036] In one possible embodiment of the present specification, the method further includes:

[0037] if the lease is obtained for the first time, obtaining an overlay ID, wherein the overlay ID is greater than the ID of all started tasks for the shared storage layer;

[0038] if the ID of the data record file with the smallest difference from the overlay ID is not less than the ID of the task record file with the smallest difference from the overlay ID, generating a data record file, marking the overlay ID for the generated data record file, and recording index information in the data record file with the smallest difference from the overlay ID in the generated data record file;

[0039] If the ID of the data record file with the smallest difference from the cover ID is smaller than the ID of the task record file with the smallest difference from the cover ID and the recorded operation execution progress is complete, a data record file is generated and the cover ID is marked for the generated data record file, and index information is recorded in the generated data record file based on the task record file with the smallest difference from the cover ID and the recorded operation execution progress is complete.

[0040] In one possible implementation of the specification, the method further includes:

[0041] receiving a data read request, obtaining the data record file with the largest ID, and reading data related to the data read request in the shared storage layer based on the index information in the data record file with the largest ID.

[0042] According to a second aspect of one or more embodiments of the specification, a data management apparatus is provided, and the apparatus includes:

[0043] an obtaining module configured to obtain an ID of a current task, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer;

[0044] an executing module configured to execute a data operation of the current task and mark the ID of the current task for data uploaded to the shared storage layer during the data operation;

[0045] a recording module configured to, if the data operation of the current task is complete, generate a data record file, mark the ID of the current task for the data record file, and record index information in the data record file, wherein the index information is used to index all data in the shared storage layer.

[0046] According to a third aspect of one or more embodiments of the specification, a computer program product is provided, including computer programs / instructions, which, when executed by a processor, implement the steps of the method of the first aspect.

[0047] According to a fourth aspect of one or more embodiments of the specification, an electronic device is provided, including:

[0048] a processor;

[0049] a memory for storing processor-executable instructions;

[0050] wherein the processor implements the method of the first aspect by running the executable instructions.

[0051] According to a fifth aspect of one or more embodiments of the present specification, a computer readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the method according to the first aspect.

[0052] The technical solutions provided by the embodiments of the present specification can include the following beneficial effects:

[0053] The data management method provided by the embodiments of the present specification obtains an ID of a current task, performs data operation of the current task, and marks the ID of the current task for data uploaded to the shared storage layer during the data operation. If the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer. Since the ID of the current task is greater than the ID of all started tasks for the shared storage layer, and the data record file for recording the index information is generated only when the data operation of the current task is completed, each version of the data record file recording the index information in the shared storage layer records the index information at the completion of the corresponding task, and the index information at the partial execution of a task is not recorded, so that the atomicity of each data task can be ensured, that is, the execution result of the task is not recorded or the execution result of the task is completely recorded; and the time sequence of the corresponding task can be determined by the ID of each version of the data record file, so that the latest data can be ensured to be read when data is read, and the service performance of the shared storage architecture is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 FIG. 1 is a flowchart of a data management method provided by an example embodiment.

[0055] Figure 2 FIG. 2 is a schematic diagram of a shared storage write node provided by an example embodiment.

[0056] Figure 3 FIG. 3 is a data directory schematic diagram of a shared storage layer provided by an example embodiment.

[0057] Figure 4 FIG. 4 is a flowchart of a data writing process provided by an example embodiment.

[0058] Figure 5 FIG. 5 is an index schematic diagram of a data reading process provided by an example embodiment.

[0059] Figure 6 FIG. 6 is a structural schematic diagram of a device provided by an example embodiment.

[0060] Figure 7 FIG. 7 is a block diagram of a data management apparatus provided by an example embodiment. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0062] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0063] First, let's explain some of the concepts involved in this manual.

[0064] Shared storage refers to storage resources that can be accessed and utilized by multiple database instances or applications simultaneously. This storage model is common in database architectures that separate storage and compute. In a shared storage architecture, all database instances share the same storage system (such as object storage, block storage, or other shared storage solutions).

[0065] SSTable (Ordered String Table): A persistent file format for storing data. Its main function is to improve read performance and write efficiency by organizing data in ordered key-value pairs on persistent storage devices.

[0066] Atomicity guarantee: For object storage that does not support the atomic operation semantics of put-if-absent (insertion fails if the object exists), write operations are not atomic and concurrent writes may occur, causing data inconsistency. Additional protocols are needed to guarantee the atomicity of read and write operations.

[0067] All database information is stored on the shared storage layer. Specifically, the database data and transaction logs are stored on the shared storage layer. Database nodes can be quickly started up using the shared storage layer, making the elastic scaling process smoother.

[0068] The database node under the shared storage architecture can read data from the shared storage layer, for example, using a local persistent storage device (such as a disk) on the database node to buffer and cache data in the shared storage layer, especially to cache data that is crucial to the performance of the TP scenario; the database node under the shared storage architecture can write data into the shared storage layer, for example, the database node can perform data modification operations such as updating cached data or writing new data, and upload the results of the data modification operations to the shared storage layer. In addition, in scenarios such as elastic scaling of the database node, restart of the database node, and merging of the ordered string table, modification and reading operations can also be performed on the data in the shared storage layer.

[0069] Some databases implement the shared storage layer based on object storage. Due to the lack of mutual exclusion capability of the object storage interface, concurrent write scenarios for the same file can occur, causing inconsistency when reading data. Therefore, how to ensure the atomicity of modification and reading operations on data in the shared storage layer, especially in typical scenarios such as outdated requests caused by network delay and failed file uploads, has become a difficult problem that needs to be solved.

[0070] The scenario of outdated requests caused by network delay is as follows: the first request is outdated due to network delay, and the second request performs modification on the data in the shared storage layer, and then the online request arrives, so that the data in the shared storage layer is restored to the old version, and the read data is not the latest data; and it is possible to read mixed data of the new and old versions, so the atomicity and consistency of the reading operation cannot be guaranteed.

[0071] The scenario of failed file upload is as follows: when the database node uploads data to the shared storage layer, part of the upload is successful and the remaining upload fails, so that the data in the shared storage layer is part of the data uploaded this time, rather than the data state before this upload or the data state of the complete upload this time, so the atomicity of the write operation cannot be guaranteed.

[0072] Based on the above technical problems, at least one embodiment of the present specification provides a data management method, which can ensure the atomicity of operations when a database node modifies, reads, or performs other operations on data in a shared storage layer, thereby ensuring the consistency of the data.

[0073] The method can be performed by a database node in a shared storage architecture, especially a shared storage architecture based on object storage. It should be understood that the data in the shared storage layer in the coexistence storage architecture can be organized in the form of a data directory, and different data is stored in the corresponding position of the data directory, thereby facilitating the management of data in the shared storage layer; of course, the data in the shared storage layer can also be organized in other data forms, and the present specification does not limit the data structure in the shared storage layer.

[0074] Please refer to the attached Figure 1 An exemplary flowchart of the data management method is shown, including steps S101 to S103.

[0075] In step S101, the ID of the current task is obtained, wherein the ID of the current task is greater than the ID of all started tasks for the shared storage layer.

[0076] The current task refers to a task to be executed at present to modify the data on the shared storage layer, for example, a task to upload data to the shared storage layer, and the uploaded data can be newly written data or data to update the existing data on the shared storage layer. The ID is a number, which is used to represent the order of the task, and the IDs of different tasks need to be sequentially increased.

[0077] Please refer to the attached Figure 2 The database node executing the method can be a shared storage writer (SS Writer) in a plurality of database nodes connected to the shared storage layer. The plurality of database nodes connected to the shared storage layer can form a log stream, and the leader node in the log stream can select a database node as the SS Writer of the log stream, preferably a database node with low load as the SS Writer of the log stream. Since there is only one database node in the log stream sharing the shared storage layer that can modify the data in the shared storage layer as the SS Writer, confusion and data errors caused by simultaneous modification of the data in the shared storage layer by different nodes are avoided.

[0078] Preferably, the above-mentioned log stream can manage the role of the SS Writer through a lease. For example, when the SS Writer is selected for the first time, the leader node of the log stream can select a database node as the SS Writer and send a lease to the node, thereby completing the allocation of the role of the SS Writer; the database node as the SS Writer initiates the renewal of the lease in the log stream when the lease is not expired (for example, when it is about to expire), thereby completing the renewal under the confirmation of other database nodes and continuing to maintain the role of the SS Writer. Of course, if the database node as the SS Writer fails, it will not initiate the renewal of the lease in the log stream, so that the leader node can reselect a database node as the SS Writer.

[0079] The database node as the SS Writer can generate a data uploading task when local data meets uploading conditions, the data uploading task can be a current task, the method can complete data uploading to the shared storage layer for the current task, and atomicity can be ensured during the uploading process and confusion with other tasks can be avoided.

[0080] For example, the data record files formed by all or part of the previous historical tasks are stored in the shared storage layer, so this step can determine the ID of the current task by traversing all the data record files stored in the shared storage layer, so that the ID of the current task is greater than the IDs of all the historical tasks. For example, the maximum ID is determined during the traversal, and the ID of the current task is obtained by incrementing the maximum ID.

[0081] For another example, the determined ID is recorded in the form of an ID record file each time the ID of the task is determined. This step can obtain the ID record file with the maximum ID, and the ID of the current task is obtained by incrementing the ID in the ID record file with the maximum ID. Then, the ID record file corresponding to the ID of the current task is constructed. For example, the maximum ID is determined by traversing all the ID record files, and the ID of the current task is obtained by incrementing the maximum ID. For example, if the maximum ID is 5, the ID of the current task can be 6.

[0082] Preferably, the shared storage layer can also have a latest ID file (i.e., a Current file) to record the ID determined for the task the last time. In the above example, the ID record file with the maximum ID can be obtained as follows: the latest ID file is obtained, and the ID record file with the maximum ID is obtained based on the ID in the latest ID file, i.e., the ID record file with the maximum ID is obtained by incrementing the ID in the latest ID file. Based on this preferred example, the ID in the latest ID file can be updated after the ID of the current task is obtained.

[0083] For example, the ID in the latest ID file is 5, and it can be determined whether there is an ID record file with the ID of 5 in the shared storage layer based on 5. If there is, it is further determined whether there is an ID record file with the ID of 5, until the ID record file with the maximum ID is obtained. For example, 6, 7, 8, …, and the ID record file with the maximum ID is determined.

[0084] Preferably, the shared storage layer can also have an oldest ID file (i.e. an oldest file) to record the earliest ID determined for a task in the remaining ID record files. For example, the data record files, ID record files, etc. in the shared storage layer that are old versions and not referenced and used can be cleared to release space and thus reduce the amount of data in the shared storage layer; the ID in the oldest ID file can be updated after each ID record file is cleared. In this way, the workload can be reduced when the largest ID record file is obtained in the step, and the efficiency of determining the ID of the session task can be improved.

[0085] It should be understood that if the shared storage layer is based on object storage, each data record file can be an object file of the object storage, each ID record file can be an object file of the object storage, the above-mentioned newest ID file can be an object file of the object storage, and the above-mentioned oldest ID file can be an object file of the object storage. That is, each time a data record file is generated, an object file is generated on the object storage as a data record file; each time an ID record file is generated, an object file is generated on the object storage as an ID record file.

[0086] Please refer to the accompanying drawings Figure 3 The data directory in the shared storage layer can be a file directory (file_directory), and a metadata directory (meta) can be provided under the file directory, and the above-mentioned data record files, ID record files, newest ID file, oldest ID file, etc. can be stored under the metadata directory. The data record file can be named and identified as sstable.list.{$op_id}, and $op_id is the ID of the task. The ID record file can be named and identified by the corresponding ID. The newest ID file can be named and identified as current. The oldest ID file can be named and identified as oldest.

[0087] In step S102, the data operation of the current task is performed, and the ID of the current task is marked for the data uploaded to the shared storage layer during the data operation.

[0088] Each time the data operation of the task is performed, the data uploaded can form an ordered string table on the shared storage layer. If the shared storage layer is based on object storage, the ordered string table formed by each task is composed of multiple object files, each object file is a segment, and the data amount of each object file is the same.

[0089] For example, if the shared storage layer is based on object storage, the step can upload the data to be uploaded of the current task to the shared storage layer in the form of an object file, and for each object file uploaded to the shared storage layer, use the ID of the current task and the upload order of the object file in the current task as the name of the object file.

[0090] The data to be uploaded of each task can include data content (i.e., data) and metadata (i.e., meta).

[0091] Preferably, when the data to be uploaded of the current task is uploaded to the shared storage layer in the form of an object file, the data content in the data to be uploaded of the current task can be uploaded to the data content directory under the shared storage layer in the form of an object file, and the metadata in the data to be uploaded of the current task can be uploaded to the metadata directory under the shared storage layer in the form of an object file.

[0092] Preferably, when the ID of the current task and the upload order of the object file in the current task are used as the name of the object file, the name of the object file can be the directory in which the object file is located, the ID of the current task, and the upload order of the object file in the current task.

[0093] Please refer to the accompanying drawings Figure 3 The data directory in the shared storage layer can be a file directory (file_directory), and the file directory can be provided with a sorted string table directory (sstable), and the sorted string table directory can be provided with a data content directory (i.e., data) and a metadata directory (meta). It should be understood that the data content directory and the metadata directory mentioned in the above preferred example can be the data content directory and the metadata directory under the sorted string table directory in the accompanying drawings Figure 3 For example, the data content in the data to be uploaded of the current task is uploaded to the data content directory under the sorted string table directory in the accompanying drawings Figure 3 in the form of an object file, and the object file is named {$op_id}_{$data_seq}, where $op_id is the ID of the current task, and data_seq is the upload order of the object file in the data content of the current task; the metadata in the data to be uploaded of the current task is uploaded to the metadata directory under the sorted string table directory in the accompanying drawings Figure 3 in the form of an object file, and the object file is named {$op_id}_{$meta_seq}, where $op_id is the ID of the current task, and meta_seq is the upload order of the object file in the metadata of the current task.

[0094] The example uploads the to-be-uploaded data of the current task to the corresponding directory of the shared storage layer in the form of an object file, and names each object file, with the ID of the current task and the order of the object file, etc. embodied in the name, so as to facilitate accurate reading of the ordered string table formed by the current task and accurate searching of the corresponding object file, i.e. the data block segment, in the ordered string table.

[0095] Further exemplarily, after obtaining the ID of the current task, a task record file can be generated, and the ID of the current task is labeled in the task record file; and during the data operation process of the current task, the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution process are recorded in the task record file.

[0096] Preferably, the operation execution record can include at least one of the following contents:

[0097] Whether completed: there are two options of 0 and 100% in the operation execution progress in this dimension, the operation execution progress is recorded as 0 when the data operation of the current task is not completed, and the operation execution progress is recorded as 100% when the data operation of the current task is completed. If the task record file is pre-provided with a finish field, the field is recorded as false when the task record file is generated; the field can be modified to true when the data operation of the current task is completed.

[0098] Upload progress: if the shared storage layer is based on object storage, the operation execution progress can also be represented according to the upload progress of the to-be-uploaded file of the current task, such as the number of uploaded object files. If the task record file is pre-provided with a max meta_seq field and a max data_seq field, the max meta_seq field is used to record the number of object files generated by the uploaded metadata of the to-be-uploaded data of the current task, and the max data_seq field is used to record the number of object files generated by the uploaded data content of the to-be-uploaded data of the current task.

[0099] Preferably, the data range of the data uploaded to the shared storage layer during the operation execution process can include:

[0100] Data range of to-be-uploaded data, i.e. data boundary of the ordered string table formed by the data uploaded by the current task. If the task record file is pre-provided with a start_scn field and an end_scn field, the start_scn field and the end_scn field are used to record the start boundary and the end boundary of the uploaded part of the to-be-uploaded data of the current task, respectively.

[0101] Preferably, the task record file can further be provided with an op_type field for recording the operation type of the current task, such as add (add, i.e. add an ordered string table), remove (remove, i.e. move an ordered string table), delete (delete, i.e. delete an ordered string table), etc.

[0102] Preferably, the task record file can further be provided with an output field. The output field is used to record the content written into the data record file corresponding to the task record file, or to record the update content of the data on the shared storage layer by the current task, i.e. the change relative to the data of the current task.

[0103] Please refer to the accompanying drawings Figure 3 The data directory in the shared storage layer can be a file directory (file_directory), and a metadata directory (meta) can be provided under the file directory, and the task record file can be stored under the metadata directory. The task record file can be named and identified as sstable.task.{$op_id}, and $op_id is the ID of the task.

[0104] It should be understood that before generating the task record file, it can also be checked whether the lease is expired, and if the lease is not expired, the task record file is generated. Thus, it can be ensured that the SS Writer having the data write permission writes data into the shared storage layer, and after the lease of the SS Writer expires, it is no longer allowed to write data into the shared storage layer, i.e. after the SS Writer switches nodes, the original node no longer has the permission to write data into the shared storage layer. Among the plurality of data nodes in communication connection with the shared storage layer, only one data node has a lease at the same time, and the data node having the lease is used to execute the current task.

[0105] It should also be understood that before recording the operation execution progress in the task record file, it can also be checked whether the lease is expired, and if the data operation of the current task is completed and the lease is not expired, the operation execution progress for indicating completion is recorded in the task record file. Thus, it can be ensured that the SS Writer having the data write permission writes data into the shared storage layer, and after the lease of the SS Writer expires, it is no longer allowed to write data into the shared storage layer, i.e. after the SS Writer switches nodes, the original node no longer has the permission to write data into the shared storage layer. Among the plurality of data nodes in communication connection with the shared storage layer, only one data node has a lease at the same time, and the data node having the lease is used to execute the current task.

[0106] In step S103, if the data operation of the current task is completed, a data record file is generated, the ID of the current task is marked in the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer.

[0107] In the method, after each execution of the data operation of the task, a data record file is formed on the shared storage layer, and index information of all data in the shared storage layer after execution of the task is recorded in the data record file.

[0108] For example, if the task record file is generated during execution of the data operation of the current task, the step can be executed as follows: if the operation execution progress recorded in the task record file is complete, a data record file is generated, the ID of the current task is marked in the data record file, and index information is recorded in the data record file.

[0109] For example, the task record file has a finish field, and the operation execution progress can be determined by checking the record of the field. If true is recorded, the operation execution progress is complete, and if false is recorded, the operation execution progress is incomplete.

[0110] For example, the task record file has an output field, and the index information can be recorded in the data record file according to the information recorded in the field. If the output field is used to record the content written in the data record file corresponding to the task record file, the record in the output field can be directly written in the data record file. If the output field is used to record the update content of the data on the shared storage layer by the current task, the index information in the data record file of the previous task (i.e., the data record file with the ID smaller than the ID of the current task and the smallest difference from the ID of the current task) can be read, and the index information written in the data record file can be obtained by superimposing the record in the output field on the basis of the index information.

[0111] For example, the index information in the data record file includes the data range (e.g., the data range represented by scn) of each ordered string table in the shared storage layer and the ID, and the ID of the ordered string table is the ID of the task written therein.

[0112] It should be understood that after determining that the data operation of the current task is completed, it can also be checked whether the lease is expired. If the data operation of the current task is completed and the lease is not expired, a data record file is generated, the ID of the current task is marked for the data record file, and index information is recorded in the data record file. Thus, it can be ensured that the SS Writer having the data write permission writes data into the shared storage layer, and the SS Writer is no longer allowed to write data into the shared storage layer after the lease of the SS Writer expires, i.e., the original node no longer has the permission to write data into the shared storage layer after the node is switched. Among them, only one data node in the plurality of data nodes in communication connection with the shared storage layer has a lease at the same time, and the data node having the lease is used to execute the current task.

[0113] It should also be understood that if the shared storage layer is constructed based on object storage, each data record file can be an object file of the object storage. That is, each time a data record file is generated, an object file is generated on the object storage as the data record file.

[0114] Please refer to the accompanying drawings Figure 3 The data directory in the shared storage layer can be a file directory (file_directory), and a metadata directory (meta) can be provided under the file directory. The above data record file can be stored under the metadata directory. The data record file can be named and identified as sstable.list.{$op_id}, and $op_id is the ID of the task.

[0115] As can be seen from the above, the accompanying drawings Figure 3 The metadata directory (meta) under the file directory (file_directory) in the accompanying drawings stores at least a latest ID file (current), an oldest ID file (oldest), at least one ID record file, a task record file of the same ID for each ID record file, and a data record file of the same ID for each ID record file. When a certain data record file is not referenced and there is a newer data record file written after it, the data record file and the ID record file and the task record file of the same ID can be deleted, and the ID recorded in the latest ID file is updated.

[0116] Please refer to the accompanying drawings Figure 4 The accompanying drawings exemplarily show the data write process obtained based on the above various embodiments. The process is written by the node as the SS Writer in the plurality of database nodes under the shared storage architecture to the shared storage layer, the shared storage layer is formed based on object storage, and the data written by the process is an ordered string table.

[0117] Step S401: Obtain the ID of the current task, write the ID record file of the ID of the current task and modify the ID in the latest ID file.

[0118] Step S402: Check the lease, and if the lease is not expired, generate the task record file of the current task.

[0119] Step S403: Perform data operation, such as uploading data of the ordered string table.

[0120] Step S404: After the data operation is completed, check the lease, and if the lease is not expired, modify the finish field in the task record file.

[0121] Step S405: Check the lease, and if the lease is not expired, generate the data record file of the current task, and make the ordered string table uploaded by the current task visible to the outside.

[0122] As can be seen from the example, the latest ID file will be overwritten, but the ID therein is not important because it only serves as the basis for obtaining the ID of the current task, and the role is to reduce the time consumption of obtaining the ID of the current task. Even if the ID is obtained and the above process is not completed and the lease is lost, the use of the ID by the database node that obtains the lease subsequently will not be affected due to the operation of checking the lease. For example, the database node that holds the lease first obtains the ID = 4 and loses the lease, and the database node that obtains the lease subsequently also obtains the ID = 4 and performs the above process. Although the database node that holds the lease first writes the ID record file of ID = 4, it cannot perform step S402 because the lease is expired, and the ID record file of ID = 4 written by the database node that holds the lease first only causes the file to be repeated and does not affect the atomicity, consistency, etc. of data writing, reading and other operations. In other words, the database node that holds the lease first does not really use the ID = 4.

[0123] The data management method provided by the embodiments of the present specification comprises the following steps: obtaining an ID of a current task; performing data operation of the current task, and marking the ID of the current task for data uploaded to the shared storage layer during the data operation; if the data operation of the current task is completed, generating a data record file and marking the ID of the current task for the data record file, and recording index information in the data record file, wherein the index information is used to index all data in the shared storage layer. Since the ID of the current task is greater than the ID of all started tasks for the shared storage layer, and the data record file for recording the index information is generated only when the data operation of the current task is completed, each version of the data record file recording the index information in the shared storage layer can record the index information at the completion of the corresponding task, and the index information at the partial execution of a task will not be recorded, so that the atomicity of each data task can be ensured, that is, the execution result of the task is not recorded or the execution result of the task is completely recorded; and the time sequence of the corresponding task can be determined by the ID of each version of the data record file, so that the latest data can be ensured to be read during data reading, and the service performance of the shared storage architecture is improved.

[0124] From the above, it can be seen that the method maintains only one database node as an SS Writer in the log stream to write data to the shared storage layer through the renewal mechanism of the lease, and adds a lease verification step in the process of writing data, such as generating a task record file, recording progress / data range, and generating a data record file, so that the write operation of the database node whose lease expires during the execution of the data write will not affect the write operation of the database node holding the lease afterwards.

[0125] In some embodiments of the present disclosure, when the SS Writer transfer occurs in the log stream, the database node newly obtaining the role of the SS writer can also check whether there is a completed data write operation of the node previously holding the SS Writer through the following steps, and if there is, a data record file is directly generated for the operation to avoid waste of computing power and reduction of efficiency caused by repeated writing:

[0126] First, if the lease is obtained for the first time, an overlay ID is obtained, wherein the overlay ID is greater than the ID of all started tasks for the shared storage layer;

[0127] Next, if the ID of the data record file with the smallest difference from the cover ID is not less than the ID of the task record file with the smallest difference from the cover ID, a data record file is generated, the cover ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the cover ID is recorded in the generated data record file; if the ID of the data record file with the smallest difference from the cover ID is less than the ID of the task record file with the smallest difference from the cover ID and the recorded operation execution progress is complete, a data record file is generated, the cover ID is marked for the generated data record file, and index information in the task record file with the smallest difference from the cover ID and the recorded operation execution progress is complete is recorded in the generated data record file.

[0128] For example, the previous database node holding the lease loses the lease after obtaining ID=4 and performing the flow to step S404 according to ID=4, and then the database node holding the subsequent lease can take ID=5 as the cover ID, generate a data record file with ID=5, and find the task record file with ID=4 and write index information in the data record file with ID=5 based on the information recorded in the output field in the task record file with ID=4 when the finish field is true. Figure 4

[0129] This embodiment can avoid repeated writing of data when a certain database node initially obtains the lease by checking the latest data record file and the task record file to obtain records that have been completed but have not yet generated data record files, thereby directly updating the data to the latest state. Moreover, the ID of the data record file generated in this process is greater than all previous existing IDs, so even if the previous ID corresponds to an operation that continues to modify the corresponding data record file or task record file, it will not affect the data record file exhibited externally. This is because, when data is read, the index information in the data record file with the largest ID is used as a reference for data reading, which will be described below.

[0130] In some embodiments of the present disclosure, any one of the database nodes under the shared storage architecture can read data in the shared storage layer according to the following steps: receiving a data read request, obtaining the data record file with the largest ID, and reading data in the shared storage layer related to the data read request based on the index information in the data record file with the largest ID.

[0131] ​For example, the data range of each ordered string table is determined according to the index information in the data record file with the largest ID, and the corresponding ordered string table is selected for data reading according to the data range of the data to be read. The ID of each ordered string table is recorded in the index information, and the metadata and data content of each ordered string table in the shared storage layer are marked with the ID (for example, the metadata and data content named by the ID and upload order in the foregoing embodiment), so that the data in the ordered string table can be searched and read according to the ID.

[0132] Please refer to the accompanying drawings Figure 5 which shows the index logic of one data reading, sequentially from the ID record file 4, the data record file sstable.list.4, the ordered string table sstable4, the table metadata meta.4_2 and the index to data.4_1, data.4_2 and data.4_3, so as to complete the partial data reading in sstable4.

[0133] As can be seen from the foregoing embodiments, the method makes the modification and reading operations of the files on the shared storage layer atomic. The operations corresponding to the old ID will not affect the data record file (i.e., the data record file with the largest ID) exhibited to the outside, that is, will not affect the atomicity of the latest reading and uploading operations, even if the corresponding data record file or task record file is continuously modified. The reading can also read the latest file uploaded in the shared storage directory. Especially in typical scenarios such as outdated requests caused by network delay, file uploading failure, etc., the atomicity of the latest reading and modification operations can be guaranteed.

[0134] Figure 6 is a schematic structural diagram of a device provided by an exemplary embodiment. Please refer to Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608 and a non-volatile memory 610, and of course can also include other hardware required by tasks. One or more embodiments of the present specification can be implemented in a software manner, such as reading the corresponding computer program from the non-volatile memory 610 into the memory 608 by the processor 602 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0135] Please refer to Figure 7 The data management apparatus can be applied to a device as shown in Figure 6 to implement the technical solutions of the present specification. The data management apparatus can include:

[0136] The obtaining module 701 is configured to obtain an ID of a current task, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer.

[0137] The executing module 702 is configured to execute a data operation of the current task, and mark the ID of the current task for data uploaded to the shared storage layer in the data operation process.

[0138] The recording module 703 is configured to generate a data record file and mark the ID of the current task for the data record file if the data operation of the current task is completed, and record index information in the data record file, wherein the index information is used to index all data in the shared storage layer.

[0139] In an embodiment of the present disclosure, the apparatus further comprises a task module configured to:

[0140] generate a task record file and mark the ID of the current task for the task record file;

[0141] record an operation execution progress and / or a data range of data uploaded to the shared storage layer in the operation execution process in the task record file in the data operation process of the current task.

[0142] In an embodiment of the present disclosure, the recording module is configured to:

[0143] generate a data record file and mark the ID of the current task for the data record file if the operation execution progress recorded in the task record file is completed, and record index information in the data record file.

[0144] In an embodiment of the present disclosure, the obtaining module is configured to:

[0145] obtain an ID record file with the largest ID, and increment the ID of the ID record file to obtain the ID of the current task;

[0146] construct an ID record file corresponding to the ID of the current task.

[0147] In an embodiment of the present disclosure, when the obtaining module is configured to obtain an ID record file with the largest ID, the obtaining module is configured to:

[0148] obtain the ID record file with the largest ID based on an ID in a latest ID file;

[0149] After obtaining the ID of the current task, the method further comprises:

[0150] updating the ID in the latest ID file.

[0151] In an embodiment of the present disclosure, the execution module is configured to perform data operations of a current task, and when data to be uploaded in the data operations is uploaded to the shared storage layer, the execution module is configured to:

[0152] upload the data to be uploaded in the current task to the shared storage layer in the form of an object file;

[0153] for each object file uploaded to the shared storage layer, use an ID of the current task and an uploading sequence of the object file in the current task as a name of the object file.

[0154] In an embodiment of the present disclosure, when the execution module is configured to upload the data to be uploaded in the current task to the shared storage layer in the form of an object file, the execution module is configured to:

[0155] upload data content in the data to be uploaded in the current task to the shared storage layer in the form of an object file under a data content directory, and upload metadata in the data to be uploaded in the current task to the shared storage layer in the form of an object file under a metadata directory;

[0156] the use of the ID of the current task and the uploading sequence of the object file in the current task as the name of the object file includes:

[0157] use a directory in which the object file is located, and the use of the ID of the current task and the uploading sequence of the object file in the current task as the name of the object file.

[0158] In an embodiment of the present disclosure, when the task module is configured to generate a task record file, the task module is configured to:

[0159] if the lease is not expired, generate the task record file;

[0160] when the task module is configured to record an operation execution progress in the task record file, the task module is configured to:

[0161] if the data operation of the current task is completed and the lease is not expired, record an operation execution progress indicating completion in the task record file;

[0162] the recording module is configured to:

[0163] if the data operation of the current task is completed and the lease is not expired, generate a data record file, mark an ID of the current task for the data record file, and record index information in the data record file;

[0164] wherein only one data node of a plurality of data nodes in communication connection with the shared storage layer has a lease at the same time, and the data node having the lease is configured to perform a current task.

[0165] In an embodiment of the present disclosure, the apparatus further includes a re-determination module configured to:

[0166] If the lease is obtained for the first time, an overlay ID is obtained, wherein the overlay ID is greater than all IDs of the started tasks for the shared storage layer;

[0167] If the ID of the data record file with the smallest difference from the overlay ID is not less than the ID of the task record file with the smallest difference from the overlay ID, a data record file is generated, the overlay ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the overlay ID is recorded in the generated data record file;

[0168] If the ID of the data record file with the smallest difference from the overlay ID is less than the ID of the task record file with the smallest difference from the overlay ID and with a recorded operation execution progress of completion, a data record file is generated, the overlay ID is marked for the generated data record file, and index information in the task record file with the smallest difference from the overlay ID and with a recorded operation execution progress of completion is recorded in the generated data record file.

[0169] In an embodiment of the present disclosure, the apparatus further includes a reading module configured to:

[0170] A data reading request is received, a data record file with the largest ID is obtained, and data related to the data reading request in the shared storage layer is read based on index information in the data record file with the largest ID.

[0171] One or more embodiments of the present disclosure further provide a computer program product, including computer programs / instructions, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0172] One or more embodiments of the present disclosure further provide a computer readable storage medium, which stores computer instructions, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0173] The system, apparatus, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0174] In a typical configuration, a computer includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0175] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) having a common memory storage port. The memory is an example of computer readable media.

[0176] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic disks storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0177] It is also important to note that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the identified element.

[0178] The above description of certain examples of the application has been presented for the purposes of illustration and description. Other examples are within the scope and range of equivalents of the claims. In some cases, acts or steps can be performed in an order different from that of the examples, and / or at least partially concurrently, and still accomplish the desired results. Also, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0179] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0180] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present description are information and data authorized by the user or authorized by all parties, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the country and region, and provide corresponding operation portal for the user to choose authorization or refusal.

[0181] It should be understood that, although the terms first, second, third, etc. can be employed in this description one or more embodiments to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one piece of information from another piece of information of the same type. For example, a first information can also be referred to as a second information, and similarly, a second information can also be referred to as a first information, without departing from the scope of one or more embodiments of the present description. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "upon" or "in response to determining".

[0182] The above descriptions are only preferred embodiments of one or more embodiments of the present description and are not intended to limit one or more embodiments of the present description. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present description shall be included in the scope of protection of one or more embodiments of the present description.

Claims

1. A data management method, the method comprising: Get the ID of the current task, where the ID of the current task is greater than the IDs of all started tasks targeting the shared storage layer; Perform data operations for the current task and label the data uploaded to the shared storage layer during the data operation with the ID of the current task; If the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked on the data record file. Index information is also recorded in the data record file, wherein the index information is used to index all data in the shared storage layer. Generate a task log file and label the task log file with the ID of the current task; During the data operation of the current task, the operation execution progress and / or the data range of data uploaded to the shared storage layer during the operation execution are recorded in the task record file; If a lease is obtained for the first time, an overlay ID is acquired, which is greater than the IDs of all initiated tasks targeting the shared storage layer. If the ID of the data record file with the smallest difference from the coverage ID is not less than the ID of the task record file with the smallest difference from the coverage ID, then a data record file is generated and the coverage ID is marked on the generated data record file, and the index information of the data record file with the smallest difference from the coverage ID is recorded in the generated data record file. If the ID of the data record file with the smallest difference from the coverage ID is less than the ID of the task record file with the smallest difference from the coverage ID and whose recorded operation execution progress is complete, then a data record file is generated and the coverage ID is marked on the generated data record file. Index information is also recorded in the generated data record file based on the task record file with the smallest difference from the coverage ID and whose recorded operation execution progress is complete.

2. The data management method according to claim 1, wherein if the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked on the data record file, and index information is recorded in the data record file, comprising: If the operation execution progress recorded in the task record file is complete, a data record file is generated, and the ID of the current task is marked on the data record file, and index information is recorded in the data record file.

3. The data management method according to claim 1, wherein obtaining the ID of the current task includes: Retrieve the ID record file with the largest ID, and increment the ID in the ID record file to obtain the ID of the current task; Construct the ID record file corresponding to the ID of the current task.

4. The data management method according to claim 3, wherein obtaining the ID record file with the largest ID includes: Obtain the latest ID file, and then obtain the ID record file with the largest ID based on the IDs in the latest ID file; After obtaining the ID of the current task, the method further includes: Update the IDs in the latest ID file.

5. The data management method according to claim 1, wherein the data operation for performing the current task, which involves labeling the data uploaded to the shared storage layer during the data operation with the ID of the current task, includes: The data to be uploaded for the current task is uploaded to the shared storage layer in the form of object files; For each object file uploaded to at least one object file in the shared storage layer, the name of the object file is determined by the ID of the current task and the upload order of the object file in the current task.

6. The data management method according to claim 5, wherein uploading the data to be uploaded for the current task to the shared storage layer in the form of an object file includes: The data content of the data to be uploaded in the current task is uploaded to the data content directory of the shared storage layer in the form of an object file, and the metadata of the data to be uploaded in the current task is uploaded to the metadata directory of the shared storage layer in the form of an object file. The step of using the ID of the current task and the upload order of the object file in the current task as the name of the object file includes: The object file is named using the directory where the object file is located, the ID of the current task, and the upload order of the object file in the current task.

7. The data management method according to claim 1, wherein generating the task record file comprises: If the lease has not expired, a task log file will be generated; Recording the operation execution progress in the task log file includes: If the data operation of the current task is completed and the lease has not expired, the progress of the completed operation is recorded in the task record file. If the data operation of the current task is completed, a data record file is generated, and the ID of the current task is marked on the data record file. Index information is also recorded in the data record file, including: If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked on the data record file, and index information is recorded in the data record file; Among the multiple data nodes that are communicatively connected to the shared storage layer, only one data node has a lease at any given time, and the data node with the lease is used to execute the current task.

8. The data management method according to claim 1, further comprising: Receive a data read request, obtain the data record file with the largest ID, and read the data related to the data read request in the shared storage layer based on the index information in the data record file with the largest ID.

9. A data management device, the device comprising: The acquisition module is used to obtain the ID of the current task, where the ID of the current task is greater than the IDs of all started tasks targeting the shared storage layer; The execution module is used to perform data operations for the current task and to label the data uploaded to the shared storage layer during the data operation with the ID of the current task; The recording module is used to generate a data recording file and mark the data recording file with the ID of the current task if the data operation of the current task is completed, and to record index information in the data recording file, wherein the index information is used to index all data in the shared storage layer; The task module is used to generate a task log file and label the task log file with the ID of the current task; during the data operation of the current task, the task log file records the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution. The re-determination module is used to: if a lease is obtained for the first time, acquire an overlay ID, wherein the overlay ID is greater than the IDs of all initiated tasks for the shared storage layer; if the ID of the data record file with the smallest difference from the overlay ID is not less than the ID of the task record file with the smallest difference from the overlay ID, generate a data record file and label the generated data record file with the overlay ID, and record index information in the generated data record file within the data record file with the smallest difference from the overlay ID; if the ID of the data record file with the smallest difference from the overlay ID is less than the ID of the task record file with the smallest difference from the overlay ID and whose recorded operation execution progress is complete, generate a data record file and label the generated data record file with the overlay ID, and record index information in the generated data record file based on the task record file with the smallest difference from the overlay ID and whose recorded operation execution progress is complete.

10. The data management device according to claim 9, wherein the recording module is used for: If the operation execution progress recorded in the task record file is complete, a data record file is generated, and the ID of the current task is marked on the data record file, and index information is recorded in the data record file.

11. The data management device according to claim 9, wherein the acquisition module is used for: Retrieve the ID record file with the largest ID, and increment the ID in the ID record file to obtain the ID of the current task; Construct the ID record file corresponding to the ID of the current task.

12. The data management device according to claim 9, wherein the execution module is used to perform data operations for the current task, and when labeling the data uploaded to the shared storage layer during the data operation with the ID of the current task, it is used to: The data to be uploaded for the current task is uploaded to the shared storage layer in the form of object files; For each object file uploaded to at least one object file in the shared storage layer, the name of the object file is determined by the ID of the current task and the upload order of the object file in the current task.

13. The data management device according to claim 9, wherein when the task module generates a task record file, it is used to: If the lease has not expired, a task log file will be generated; The task module is used to record the operation execution progress in the task log file for the following purposes: If the data operation of the current task is completed and the lease has not expired, the progress of the completed operation is recorded in the task record file. The recording module is used for: If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked on the data record file, and index information is recorded in the data record file; in, Of the multiple data nodes that are communicatively connected to the shared storage layer, only one data node has a lease at any given time, and the data node with the lease is used to execute the current task.

14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

15. An electronic device comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 8 by executing the executable instructions.

16. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 8.

Images