Data management method and device, electronic equipment and storage medium

Abstract

The invention provides a data management method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the ID of a current task, the ID of the current task being greater than the IDs of all started tasks for a shared storage layer; executing the data operation of the current task, and labeling the ID of the current task for the data uploaded to the shared storage layer in the data operation process; if the data operation of the current task is completed, a data recording file is generated, the ID of the current task is marked for the data recording file, index information is recorded in the data recording file, and the index information is used for indexing all data in the shared storage layer.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a data management method and device, an electronic device, and a storage medium. Background Art

[0002] With the exponential growth of data in the computer field and the development of database technology, shared storage has gradually emerged. That is, by setting up a shared storage layer and multiple database nodes that communicate with it, user data can be stored in the shared storage layer for sharing by multiple database nodes. The database nodes will cache part of the data in the shared storage layer locally and provide data services to the outside world based on it.

[0003] The database node also needs to upload data processing results, such as data writing results, data modification results, etc., to the shared storage layer to update the data in the shared storage layer. However, in the related art, it is difficult for database nodes under the shared storage architecture to ensure the atomicity of data operations, which causes confusion or even errors in the data in the shared storage layer. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a data management method and device, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a data management method is proposed, the method comprising: Get the ID of the current task, where the ID of the current task is greater than the IDs of all started tasks for the shared storage layer; Execute the data operation of the current task, and mark the ID of the current task for the data uploaded to the shared storage layer during the data operation; If the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer.

[0006] In a possible embodiment of this specification, the method further includes: Generate a task record file and mark the task record file with the ID of the current task; During the data operation process of executing the current task, the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution process are recorded in the task record file.

[0007] In a possible embodiment of the present specification, if the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, including: If the operation execution progress recorded in the task record file is completed, a data record file is generated and the ID of the current task is marked in the data record file, and index information is recorded in the data record file.

[0008] In a possible embodiment of this specification, obtaining the ID of the current task includes: Get the ID record file with the largest ID, and increment it based on the ID of the ID record file to get the ID of the current task; Build the ID record file corresponding to the ID of the current task.

[0009] In a possible embodiment of the present specification, obtaining the ID record file with the largest ID includes: Obtain the latest ID file, and obtain the ID record file with the largest ID based on the ID in the latest ID file; After obtaining the ID of the current task, the method further includes: Update the ID in the latest ID file.

[0010] In a possible embodiment of the present specification, the data operation of executing the current task, marking the ID of the current task for the data uploaded to the shared storage layer during the data operation, includes: Upload the data to be uploaded of the current task to the shared storage layer in the form of an object file; For each object file in the at least one object file uploaded to the shared storage layer, the ID of the current task and the upload order of the object file in the current task are used as the name of the object file.

[0011] In a possible embodiment of the present specification, uploading the to-be-uploaded data of the current task to the shared storage layer in the form of an object file includes: Upload the data content of the data to be uploaded of the current task to the data content directory of the shared storage layer in the form of an object file, and upload the metadata of the data to be uploaded of the current task to the metadata directory of the shared storage layer in the form of an object file; The method of using the ID of the current task and the upload order of the object file in the current task as the name of the object file includes: The name of the object file is the directory where the object file is located, the ID of the current task, and the upload order of the object file in the current task.

[0012] In a possible embodiment of this specification, the generating of the task record file includes: If the lease has not expired, a task record file is generated; The recording of the operation execution progress in the task record file includes: If the data operation of the current task is completed and the lease has not expired, the progress of the operation execution used to indicate the completion is recorded in the task record file; If the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, including: If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file; Among the multiple data nodes that are communicatively connected to the shared storage layer, only one data node has a lease at the same time, and the data node with the lease is used to execute the current task.

[0013] In a possible embodiment of this specification, the method further includes: If the lease is obtained for the first time, the coverage ID is obtained, where the coverage ID is greater than the IDs of all started tasks for the shared storage layer; If the ID of the data record file with the smallest difference from the coverage ID is not less than the ID of the task record file with the smallest difference from the coverage ID, a data record file is generated and the coverage ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the coverage ID is recorded in the generated data record file; If the ID of the data record file with the smallest difference with the overwrite ID is smaller than the ID of the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed, a data record file is generated and the generated data record file is marked with the overwrite ID, and index information is recorded in the generated data record file based on the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed.

[0014] In a possible embodiment of this specification, the method further includes: A data read request is received, a data record file with a largest ID is obtained, and data related to the data read request in the shared storage layer is read based on index information in the data record file with the largest ID.

[0015] According to a second aspect of one or more embodiments of this specification, a data management device is provided, the device comprising: An acquisition module, used to acquire the ID of the current task, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer; An execution module, used to execute the data operation of the current task and mark the ID of the current task for the data uploaded to the shared storage layer during the data operation; The recording module is used to generate a data recording file and mark the ID of the current task for the data recording file if the data operation of the current task is completed, and record index information in the data recording file, wherein the index information is used to index all data in the shared storage layer.

[0016] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0017] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor implements the method as described in the first aspect by running the executable instructions.

[0018] According to a fifth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] The technical solutions provided by the embodiments of this specification may have the following beneficial effects: The data management method provided in the embodiment of this specification obtains the ID of the current task; performs the data operation of the current task, and marks the ID of the current task for the data uploaded to the shared storage layer during the data operation; if the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer. Since the ID of the current task is greater than the ID of all the started tasks for the shared storage layer, and the data record file for recording the index information is generated only when the data operation of the current task is completed, each version of the data record file that records the index information in the shared storage layer can record the index information when the corresponding task is completed, and there will be no index information recorded when a certain task is partially executed, so that each data task can ensure atomicity, that is, the execution result of the task is not recorded at all or the execution result of the task is completely completed; and each version of the data record file can judge the timing of the corresponding task by the ID, so that the latest data can be read when the data is read, thereby improving the service performance of the shared storage architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a data management method provided by an exemplary embodiment.

[0021] Figure 2 is a schematic diagram of a shared storage write node provided by an exemplary embodiment.

[0022] Figure 3 It is a schematic diagram of a data directory of a shared storage layer provided by an exemplary embodiment.

[0023] Figure 4 is a flow chart of a data writing process provided by an exemplary embodiment.

[0024] Figure 5 It is an index diagram of a data reading process provided by an exemplary embodiment.

[0025] Figure 6 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0026] Figure 7 is a block diagram of a data management device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] First, some concepts involved in this specification are explained.

[0030] Shared storage: refers to storage resources that can be accessed and used by multiple database instances or applications at the same time. This storage model is common in the storage and computing separation database architecture. In the shared storage architecture, all database instances share the same storage system (such as object storage, block storage, or other shared storage solutions).

[0031] Ordered string table SSTable: A persistent file format for storing data. Its main function is to improve read performance and write efficiency by organizing data in the form of ordered key-value pairs on persistent storage devices.

[0032] Atomicity guarantee: For object storage that does not support put-if-absent atomic operation semantics (insert fails if the object exists), write operations are not atomic, and concurrent writes may occur, resulting in inconsistent data reading. Additional protocols are needed to ensure the atomicity of read and write operations.

[0033] The shared storage layer has all the database information. Specifically, the database data and transaction logs are stored on the shared storage layer. Database nodes can use the shared storage layer to quickly pull up, making the elastic expansion and contraction process smoother.

[0034] Database nodes under the shared storage architecture can read data from the shared storage layer. For example, local persistent storage devices (such as disks) are used on database nodes to buffer and cache data in the shared storage layer, especially data that is critical to the performance of TP scenarios. Database nodes under the shared storage architecture can write data to the shared storage layer. For example, database nodes can perform data modification operations such as updating cached data or writing new data, and upload the results of data modification operations to the shared storage layer. In addition, in scenarios such as elastic expansion and contraction of database nodes, restart after downtime, and merging of ordered string tables, operations such as modification and reading can also be performed on data in the shared storage layer.

[0035] Some databases use object storage to implement a shared storage layer. Since the object storage interface lacks mutual exclusion capabilities, there may be scenarios where concurrent writes to the same file occur, causing inconsistent data reads. Therefore, how to ensure the atomicity of modification and reading operations on data in the shared storage layer, especially in typical scenarios such as outdated requests caused by network delays and file upload failures, has become a difficult problem that needs to be solved urgently.

[0036] The scenario where network delays lead to outdated requests is as follows: the first request causes network delays and becomes outdated, and the online request arrives after the subsequent request modifies the data in the shared storage layer, causing the data in the shared storage layer to be restored to the old version. In this case, the data read is not the latest data; and it is possible that a mixture of new and old versions of data is read, so the atomicity and data consistency of the read operation cannot be guaranteed.

[0037] The scenarios where file upload fails are as follows: when the database node uploads data to the shared storage layer, some uploads succeed but the rest fail to upload, resulting in the data in the shared storage layer being partial data uploaded this time, rather than the data status before this upload, or the data status of this complete upload, so the atomicity of the write operation cannot be guaranteed.

[0038] Based on the above technical problems, at least one embodiment of this specification provides a data management method, which can ensure the atomicity of operations when the database node modifies, reads, and other operations on data in the shared storage layer, thereby ensuring data consistency.

[0039] The method can be executed by a shared storage architecture, especially a database node in a shared storage architecture built based on object storage. It should be understood that the data in the shared storage layer in the coexistence storage architecture can be organized in the form of a data directory, and different data is stored in corresponding positions of the data directory, so as to facilitate data management in the shared storage layer; of course, the data in the coexistence storage layer can also be organized in other data forms, and this specification does not limit the data structure in the shared storage layer.

[0040] Please refer to the attached Figure 1 , which exemplarily shows a flow chart of the data management method, including steps S101 to S103.

[0041] In step S101, the ID of the current task is obtained, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer.

[0042] The current task refers to the task to be executed to modify the data on the shared storage layer, such as the task of uploading data to the shared storage layer. The uploaded data can be newly written data or data that updates the existing data on the shared storage layer. ID is a number used to represent the order of tasks. The IDs of different tasks must increase in order.

[0043] Please refer to the attached Figure 2 , the database node executing the method can be a shared storage writing node SS Writer (Shared Storage Writer) among the multiple database nodes connected to the shared storage layer. The multiple database nodes connected to the shared storage layer can form a log stream, and the master node (leader node) in the log stream can select a database node as the SS Writer of the log stream, preferably a database node with a lower load as the SS Writer of the log stream. Since only one database node in the log stream of the shared storage layer can act as an SS Writer to modify the data in the shared storage layer, confusion and data errors caused by different nodes modifying the data in the shared storage layer at the same time can be avoided.

[0044] Preferably, the above-mentioned log stream can manage the role of SS Writer through a lease. For example, when selecting an SS Writer for the first time, the master node of the log stream can select a database node as the SS Writer and send the lease to the node, thereby completing the allocation of the role of SS Writer; the database node serving as the SS Writer initiates the renewal of the lease in the log stream when the lease has not expired (for example, when it is about to expire), thereby completing the renewal with the confirmation of other database nodes and continuing to maintain the role of SS Writer. Of course, if the database node serving as the SS Writer fails, the renewal of the lease will not be initiated in the log stream, so that the leader node can reselect a database node as the SS Writer.

[0045] When the local data of the database node serving as SS Writer meets the upload conditions, a data upload task can be generated. The data upload task can be used as the current task. This method can complete the upload of data to the shared storage layer for the current task, and atomicity can be guaranteed during the upload process without confusion with other tasks.

[0046] Exemplarily, the data record files formed by all or part of the previous historical tasks are stored in the shared storage layer, so this step can determine the ID of the current task by traversing all the data record files stored in the shared storage layer, so that the ID of the current task is greater than the IDs of all historical tasks. For example, the maximum ID is determined during the traversal process, and the maximum ID is incremented to obtain the ID of the current task.

[0047] As another example, each time the ID of a task is determined, the determined ID is recorded in the form of an ID record file. In this step, the ID record file with the largest ID can be obtained, and the ID of the current task can be obtained by incrementing the ID of the ID record file; then, the ID record file corresponding to the ID of the current task is constructed. For example, the maximum ID can be determined by traversing all ID record files, and then the ID of the current task can be obtained by incrementing the maximum ID. If the maximum ID is 5, the ID of the current task can be 6.

[0048] Preferably, the shared storage layer may also have a latest ID file (i.e., Current file) to record the latest ID determined for the task. In the above example, obtaining the ID record file with the largest ID can be performed as follows: obtaining the latest ID file, and obtaining the ID record file with the largest ID based on the ID in the latest ID file, that is, obtaining the ID record file with the largest ID by incrementing the ID in the latest ID file. Based on this preferred example, after obtaining the ID of the current task, the ID in the latest ID file can be updated.

[0049] For example, if the ID in the latest ID file is 5, then it can be determined based on 5 whether there is an ID record file with ID 5 in the shared storage layer, and if so, it can be further determined whether there is an ID record file with ID 5, until the ID record file with the largest ID is obtained. For example, 6, 7, 8, ... are determined in sequence until the ID record file with the largest ID is determined.

[0050] Preferably, the shared storage layer may also have an earliest ID file (i.e., oldest file) to record the earliest ID determined for the task in the currently remaining ID record files. For example, old versions of data record files and ID record files that are not referenced and used in the shared storage layer can be cleared to free up space, thereby reducing the amount of data in the shared storage layer; each time the ID record file is cleared, the ID in the oldest ID file can be updated. In this way, the workload can be reduced when obtaining the largest ID record file in this step, and the efficiency of determining the ID of the schedule task can be improved.

[0051] It should be understood that if the shared storage layer is built based on object storage, each data record file can be an object file of the object storage, each ID record file can be an object file of the object storage, the latest ID file can be an object file of the object storage, and the earliest ID file can be an object file of the object storage. That is, each time a data record file is generated, an object file is generated on the object storage as a data record file; each time an ID record file is generated, an object file is generated on the object storage as an ID record file.

[0052] Please refer to the attached Figure 3 , the data directory in the shared storage layer can be a file directory (file_directory), and a metadata directory (meta) can be set under the file directory. The above-mentioned data record files, ID record files, latest ID files, and earliest ID files can be stored under the metadata directory. The data record file can be named and identified with sstable.list.{$op_id}, where $op_id is the ID of the task. The ID record file can be named and identified with the corresponding ID. The latest ID file can be named and identified with current. The earliest ID file can be named and identified with oldest.

[0053] In step S102, data operation of the current task is performed, and the data uploaded to the shared storage layer during the data operation is marked with the ID of the current task.

[0054] The data uploaded by the data operation of each task can form an ordered string table on the shared storage layer. If the shared storage layer is built based on object storage, the ordered string table formed by each task consists of multiple object files, each object file is a block segment, and the data volume of each object file is the same.

[0055] Exemplarily, if the shared storage layer is built based on object storage, this step can upload the data to be uploaded of the current task to the shared storage layer in the form of an object file; and for each object file in at least one object file uploaded to the shared storage layer, the ID of the current task and the upload order of the object file in the current task are used as the name of the object file.

[0056] The data to be uploaded of each task may include data content (ie, data) and metadata (ie, meta), and the metadata in the data to be uploaded is the metadata of the data to be uploaded.

[0057] Preferably, when the data to be uploaded of the current task is uploaded to the shared storage layer in the form of an object file, the data content in the data to be uploaded of the current task can be uploaded to the data content directory of the shared storage layer in the form of an object file, and the metadata in the data to be uploaded of the current task can be uploaded to the metadata directory of the shared storage layer in the form of an object file.

[0058] Preferably, when the ID of the current task and the order in which the object file is uploaded in the current task are used as the name of the object file, the directory where the object file is located, the ID of the current task and the order in which the object file is uploaded in the current task can be used as the name of the object file.

[0059] Please refer to the attached Figure 3 , the data directory in the shared storage layer can be a file directory (file_directory), under which there can be an ordered string table directory (sstable), under which there are a data content directory (data) and a metadata directory (meta). It should be understood that the data content directory and metadata directory mentioned in the above preferred example can be attached Figure 3 The data content directory and metadata directory under the ordered string table directory in the current task are uploaded to the attached file in the form of an object file. Figure 3 The data content directory under the ordered string table directory in the task, the object file is named {$op_id}_{$data_seq}, where $op_id is the ID of the current task, and data_seq is the upload order of the object file in the data content of the current task; the metadata in the data to be uploaded of the current task is uploaded to the attached file in the form of an object file. Figure 3 In the metadata directory under the ordered string table directory, the object files are named {$op_id}_{$meta_seq}, where $op_id is the ID of the current task and meta_seq is the upload order of the object files in the metadata of the current task.

[0060] This example uploads the data to be uploaded of the current task to the corresponding directory of the shared storage layer in the form of object files, and names each object file. The name reflects the ID of the current task and the order of the object files, so that it is easy to accurately find the ordered string table formed by the current task when reading, and accurately find the corresponding object file in the ordered string table, that is, the data block segment, etc.

[0061] As another example, after obtaining the ID of the current task, a task record file can be generated and the ID of the current task can be marked in the task record file; and in the process of executing the data operation of the current task, the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution process are recorded in the task record file.

[0062] Preferably, the operation execution record may include at least one of the following contents: Completed: There are two options for the operation execution progress in this dimension: 0 and 100%. If the data operation of the current task is not completed, the operation execution progress is recorded as 0; after the data operation of the current task is completed, the operation execution progress is recorded as 100%. If there is a finish field preset in the task record file, this field is recorded as false when the task record file is generated; you can modify this field to true after the data operation of the current task is completed.

[0063] Upload progress: If the shared storage layer is built based on object storage, the operation execution progress can also be characterized according to the upload progress of the files to be uploaded of the current task, such as by the number of object files uploaded. For example, the task record file is preset with a max_meta_seq field and a max_data_seq field, the max_meta_seq field is used to record the number of object files generated by the metadata of the uploaded data to be uploaded of the current task, and the max_data_seq field is used to record the number of object files generated by the data content of the uploaded data to be uploaded of the current task.

[0064] Preferably, the data range of the data uploaded to the shared storage layer during the execution of the operation may include: The data range of the data to be uploaded, that is, the data boundary of the ordered string table formed by the data uploaded by the current task. For example, the task record file is preset with the start_scn field and the end_scn field, which are used to record the start boundary and end boundary of the uploaded part of the data to be uploaded of the current task respectively.

[0065] Preferably, the task record file may also have an op_type field for recording the operation type of the current task, such as add (i.e., adding an ordered string table), remove (i.e., moving an ordered string table), delete (i.e., deleting an ordered string table), etc.

[0066] Preferably, the task record file may also be provided with an output field. The output field is used to record the content of the data record file corresponding to the written task record file, or to record the update content of the current task on the data on the shared storage layer, that is, the change of the data relative to the current task.

[0067] Please refer to the attached Figure 3 , the data directory in the shared storage layer can be a file directory (file_directory), and a metadata directory (meta) can be set under the file directory. The above task record file can be stored under the metadata directory. The task record file can be named and identified with sstable.task.{$op_id}, where $op_id is the ID of the task.

[0068] It should be understood that before generating the task record file, it is also possible to check whether the lease has expired. If the lease has not expired, the task record file is generated. This ensures that the SS Writer with data write permission writes data into the shared storage layer, and that the SS Writer is no longer allowed to write data into the shared storage layer after the lease expires, that is, after the SS Writer switches nodes, the original node no longer has the permission to write data into the shared storage layer. Among them, only one data node among the multiple data nodes that are connected to the shared storage layer at the same time has a lease, and the data node with the lease is used to execute the current task.

[0069] It should also be understood that before recording the progress of the operation execution in the task record file, it is also possible to check whether the lease has expired. If the data operation of the current task is completed and the lease has not expired, the progress of the operation execution used to characterize the completion is recorded in the task record file. This ensures that the SS Writer with data write permission writes data to the shared storage layer, and that the SS Writer is no longer allowed to write data to the shared storage layer after the lease expires, that is, after the SS Writer switches nodes, the original node no longer has the permission to write data to the shared storage layer. Among them, only one data node among the multiple data nodes that are connected to the shared storage layer in communication at the same time has a lease, and the data node with the lease is used to execute the current task.

[0070] In step S103, if the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer.

[0071] After each data operation of a task is executed, a data record file may be formed on the shared storage layer to record index information of all data in the shared storage layer after the task is executed.

[0072] Exemplarily, if a task record file is generated during the execution of the data operation of the current task, this step can be performed as follows: if the operation execution progress recorded in the task record file is completed, a data record file is generated and the ID of the current task is marked in the data record file, and index information is recorded in the data record file.

[0073] For example, if there is a finish field in the task record file, you can check the record of this field. If the record contains true, the operation execution progress is completed; if the record contains false, the operation execution progress is incomplete.

[0074] For example, if there is an output field in the task record file, index information can be recorded in the data record file according to the information recorded in the field. If the output field is used to record the content in the data record file corresponding to the task record file, the record in the output field can be directly written into the data record file; if the output field is used to record the update content of the current task on the shared storage layer, the index information in the data record file of the previous task (that is, the data record file whose ID is smaller than the ID of the current task and whose difference with the ID of the current task is the smallest) can be read, and the record in the output field can be superimposed on the index information to obtain the index information written into the data record file.

[0075] For example, the index information in the data record file includes the data range (eg, data range represented by scn) and ID of each ordered string table in the shared storage layer, and the ID of the ordered string table is the ID of the task that writes it.

[0076] It should be understood that after determining that the data operation of the current task is completed, it is also possible to check whether the lease has expired. If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file. This ensures that the SS Writer with data write permission writes data to the shared storage layer, and that the SS Writer is no longer allowed to write data to the shared storage layer after the lease expires, that is, after the SS Writer switches nodes, the original node no longer has the permission to write data to the shared storage layer. Among them, only one data node among the multiple data nodes that are connected to the shared storage layer at the same time has a lease, and the data node with the lease is used to execute the current task.

[0077] It should also be understood that if the shared storage layer is constructed based on object storage, each data record file can be an object file of the object storage, that is, each time a data record file is generated, an object file is generated on the object storage as the data record file.

[0078] Please refer to the attached Figure 3 , the data directory in the shared storage layer can be a file directory (file_directory), and a metadata directory (meta) can be set under the file directory. The above data record files can be stored under the metadata directory. The data record files can be named and identified with sstable.list.{$op_id}, where $op_id is the ID of the task.

[0079] In summary, the attached Figure 3 The metadata directory (meta) under the file directory (file_directory) stores at least the latest ID file (current), the earliest ID file (oldest), at least one ID record file, each task record file with the same ID as the ID record file, and each data record file with the same ID as the ID record file; when a data record file is not referenced and a newer data record file is written, the data record file and the ID record file and task record file with the same ID can be deleted, and the ID recorded in the latest ID file can be updated.

[0080] Please refer to the attached Figure 4 , which exemplarily shows the data writing process obtained based on the above-mentioned embodiments, in which a node serving as an SS Writer among multiple database nodes under a shared storage architecture writes data into a shared storage layer, and the shared storage layer is formed based on object storage. The data written in this process is an ordered string table.

[0081] Step S401: Get the ID of the current task, write the ID of the current task into the ID record file and modify the ID in the latest ID file; Step S402: Check the lease, and if the lease has not expired, generate a task record file for the current task.

[0082] Step S403: performing data operations, such as uploading data of the ordered string table.

[0083] Step S404: After the data operation is completed, the lease is checked. If the lease has not expired, the finish field in the task record file is modified.

[0084] Step S405: Check the lease. If the lease has not expired, generate a data record file for the current task and make the ordered string table uploaded by the current task visible to the outside.

[0085] As can be seen from this example, although the latest ID file will be overwritten, the ID in it is not important, because it only serves as the basis for obtaining the ID of the current task, and its role is to reduce the time spent on obtaining the ID of the current task; even if the lease is lost after a certain ID is obtained but the above process is not completed, the operation of checking the lease will not affect the use of the ID by the database node that subsequently obtains the lease. For example, the previous database node that holds the lease loses the lease after obtaining ID=4, and the database node that subsequently obtains the lease also obtains ID=4 and executes the above process. Although the previous database node that holds the lease will subsequently write the ID record file of ID=4, it cannot execute step S402 due to the expiration of the lease, and the ID record file of ID=4 written by it only causes the file to be repeated, and does not affect the atomicity and consistency of operations such as data writing and reading. In other words, the previous database node that holds the lease does not actually use ID=4.

[0086] The data management method provided in the embodiment of this specification obtains the ID of the current task; performs the data operation of the current task, and marks the ID of the current task for the data uploaded to the shared storage layer during the data operation; if the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer. Since the ID of the current task is greater than the ID of all the started tasks for the shared storage layer, and the data record file for recording the index information is generated only when the data operation of the current task is completed, each version of the data record file that records the index information in the shared storage layer can record the index information when the corresponding task is completed, and there will be no index information recorded when a certain task is partially executed, so that each data task can ensure atomicity, that is, the execution result of the task is not recorded at all or the execution result of the task is completely completed; and each version of the data record file can judge the timing of the corresponding task by the ID, so that the latest data can be read when the data is read, thereby improving the service performance of the shared storage architecture.

[0087] From the above content, it can be seen that this method uses the lease renewal mechanism to maintain only one database node in the log stream as the SS Writer to write data to the shared storage layer, and adds lease verification steps to the steps of generating task record files, recording progress / data range, and generating data record files in the process of writing data. In this way, the write operation of the database node whose lease expires during the data writing process will not affect the write operation of the database node that holds the lease later.

[0088] In some embodiments of the present disclosure, when SS Writer transfer occurs in the log stream, the database node that newly obtains the SS writer role can also check whether the node that previously held the SS Writer has a completed data write operation through the following steps. If so, a data record file is directly generated for the operation to avoid repeated writing resulting in waste of computing power and reduced efficiency: First, if the lease is obtained for the first time, the coverage ID is obtained, where the coverage ID is greater than the IDs of all started tasks for the shared storage layer; Next, if the ID of the data record file with the smallest difference from the overwrite ID is not less than the ID of the task record file with the smallest difference from the overwrite ID, a data record file is generated and the overwrite ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the overwrite ID is recorded in the generated data record file; if the ID of the data record file with the smallest difference from the overwrite ID is less than the ID of the task record file with the smallest difference from the overwrite ID and the recorded operation execution progress is completed, a data record file is generated and the overwrite ID is marked for the generated data record file, and index information is recorded in the generated data record file based on the task record file with the smallest difference from the overwrite ID and the recorded operation execution progress is completed.

[0089] For example, the previous database node holding the lease obtains ID = 4 and executes the attached Figure 4 If the lease is lost after the process reaches step S404, the database node that subsequently obtains the lease can use ID=5 as the overwriting ID, generate a data record file with ID=5, and when the task record file with ID=4 is found and its finish field is true, write index information to the data record file with ID=5 based on the information recorded in the output field in the task record file with ID=4.

[0090] When a database node first obtains a lease, this embodiment checks the latest data record file and task record file to obtain records of data record files that have been completed but not yet generated, thereby avoiding repeated writing of data and directly updating the data to the latest state; and the ID of the data record file generated by this process is larger than all previous existing IDs, so even if the operation corresponding to the previous ID continues to modify the corresponding data record file or task record file, it will not affect the data record file displayed to the outside. This is because when reading data, the index information in the data record file with the largest ID is used as a reference for data reading, which will be introduced below.

[0091] In some embodiments of the present disclosure, any database node under the shared storage architecture can read the data in the shared storage layer according to the following steps: receive a data read request, obtain the data record file with the largest ID, and read the data related to the data read request in the shared storage layer based on the index information in the data record file with the largest ID.

[0092] For example, the data range of each ordered string table is determined based on the index information in the data record file with the largest ID, and the corresponding ordered string table is selected for data reading based on the data range of the data to be read. The index information records the ID of each ordered string table, and the metadata and data content of each ordered string table in the shared storage layer are marked with an ID (for example, the metadata and data content named by the ID and the upload order in the aforementioned embodiment), so the data in the ordered string table can be searched and read accordingly.

[0093] Please refer to the attached Figure 5 , which shows the index logic of a data read, indexed in sequence from ID record file 4, data record file sstable.list.4, ordered string table sstable4, metadata meta.4_2 in the table to data.4_1, data.4_2, data.4_3, thereby completing the reading of part of the data in sstable4.

[0094] It can be seen from the above embodiments that this method makes the modification and reading operations of files on the shared storage layer atomic. Even if the operation corresponding to the old ID continues to modify the corresponding data record file or task record file, it will not affect the data record file displayed to the outside (that is, the data record file with the largest ID), that is, it will not affect the atomicity of the latest read and upload operations. Reading can also read the latest uploaded files in the shared storage directory. In particular, the atomicity of the latest read and modify operations can be guaranteed in typical scenarios such as outdated requests caused by network delays and file upload failures.

[0095] Figure 6 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0096] Please refer to Figure 7 , the data management device can be used for Figure 6 The data management device may include: An acquisition module 701 is used to acquire the ID of the current task, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer; An execution module 702 is used to execute the data operation of the current task and mark the ID of the current task for the data uploaded to the shared storage layer during the data operation; The recording module 703 is used to generate a data recording file and mark the ID of the current task for the data recording file if the data operation of the current task is completed, and record index information in the data recording file, wherein the index information is used to index all data in the shared storage layer.

[0097] In one embodiment of the present specification, the device further includes a task module, which is used to: Generate a task record file and mark the task record file with the ID of the current task; During the data operation process of executing the current task, the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution process are recorded in the task record file.

[0098] In one embodiment of this specification, the recording module is used to: If the operation execution progress recorded in the task record file is completed, a data record file is generated and the ID of the current task is marked in the data record file, and index information is recorded in the data record file.

[0099] In one embodiment of this specification, the acquisition module is used to: Get the ID record file with the largest ID, and increment it based on the ID of the ID record file to get the ID of the current task; Build the ID record file corresponding to the ID of the current task.

[0100] In one embodiment of the present specification, when the acquisition module is used to acquire the ID record file with the largest ID, it is used to: Obtain the latest ID file, and obtain the ID record file with the largest ID based on the ID in the latest ID file; After obtaining the ID of the current task, the method further includes: Update the ID in the latest ID file.

[0101] In one embodiment of the present specification, the execution module is used to execute the data operation of the current task, and when marking the ID of the current task for the data uploaded to the shared storage layer during the data operation, it is used to: Upload the data to be uploaded of the current task to the shared storage layer in the form of an object file; For each object file in the at least one object file uploaded to the shared storage layer, the ID of the current task and the upload order of the object file in the current task are used as the name of the object file.

[0102] In one embodiment of the present specification, when the execution module is used to upload the data to be uploaded of the current task to the shared storage layer in the form of an object file, it is used to: Upload the data content of the data to be uploaded of the current task to the data content directory of the shared storage layer in the form of an object file, and upload the metadata of the data to be uploaded of the current task to the metadata directory of the shared storage layer in the form of an object file; The method of using the ID of the current task and the upload order of the object file in the current task as the name of the object file includes: The name of the object file is the directory where the object file is located, the ID of the current task, and the upload order of the object file in the current task.

[0103] In one embodiment of the present specification, when the task module is used to generate a task record file, it is used to: If the lease has not expired, a task record file is generated; The task module is used to record the operation execution progress in the task record file, and is used to: If the data operation of the current task is completed and the lease has not expired, the progress of the operation execution used to indicate the completion is recorded in the task record file; The recording module is used to: If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file; Among the multiple data nodes that are communicatively connected to the shared storage layer, only one data node has a lease at the same time, and the data node with the lease is used to execute the current task.

[0104] In one embodiment of the present specification, the apparatus further includes a re-determination module, configured to: If the lease is obtained for the first time, the coverage ID is obtained, where the coverage ID is greater than the IDs of all started tasks for the shared storage layer; If the ID of the data record file with the smallest difference from the coverage ID is not less than the ID of the task record file with the smallest difference from the coverage ID, a data record file is generated and the coverage ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the coverage ID is recorded in the generated data record file; If the ID of the data record file with the smallest difference with the overwrite ID is smaller than the ID of the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed, a data record file is generated and the generated data record file is marked with the overwrite ID, and index information is recorded in the generated data record file based on the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed.

[0105] In one embodiment of the present specification, the device further includes a reading module, which is used to: A data read request is received, a data record file with a largest ID is obtained, and data related to the data read request in the shared storage layer is read based on index information in the data record file with the largest ID.

[0106] One or more embodiments of the present specification also propose a computer program product, including a computer program / instruction, which implements the steps of the method provided in the first aspect when the computer program / instruction is executed by a processor.

[0107] One or more embodiments of the present specification also propose a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0108] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0109] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0110] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0111] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0113] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0115] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0116] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0117] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data management method, the method comprising: Get the ID of the current task, where the ID of the current task is greater than the IDs of all started tasks for the shared storage layer; Execute the data operation of the current task, and mark the ID of the current task for the data uploaded to the shared storage layer during the data operation; If the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, wherein the index information is used to index all data in the shared storage layer.

2. The data management method according to claim 1, further comprising: Generate a task record file and mark the task record file with the ID of the current task; During the data operation process of executing the current task, the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution process are recorded in the task record file.

3. The data management method according to claim 2, wherein if the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, including: If the operation execution progress recorded in the task record file is completed, a data record file is generated and the ID of the current task is marked in the data record file, and index information is recorded in the data record file.

4. The data management method according to claim 1, wherein obtaining the ID of the current task comprises: Get the ID record file with the largest ID, and increment it based on the ID of the ID record file to get the ID of the current task; Build the ID record file corresponding to the ID of the current task.

5. The data management method according to claim 4, wherein the step of obtaining the ID record file with the largest ID comprises: Obtain the latest ID file, and obtain the ID record file with the largest ID based on the ID in the latest ID file; After obtaining the ID of the current task, the method further includes: Update the ID in the latest ID file.

6. The data management method according to claim 1, wherein the data operation of executing the current task is to mark the ID of the current task for the data uploaded to the shared storage layer during the data operation, and comprises: Upload the data to be uploaded of the current task to the shared storage layer in the form of an object file; For each object file in the at least one object file uploaded to the shared storage layer, an ID of a current task and an upload order of the object file in the current task are used as the name of the object file.

7. The data management method according to claim 6, wherein uploading the data to be uploaded of the current task to the shared storage layer in the form of an object file comprises: Upload the data content of the data to be uploaded of the current task to the data content directory of the shared storage layer in the form of an object file, and upload the metadata of the data to be uploaded of the current task to the metadata directory of the shared storage layer in the form of an object file; The method of using the ID of the current task and the upload order of the object file in the current task as the name of the object file includes: The name of the object file is the directory where the object file is located, the ID of the current task, and the upload order of the object file in the current task.

8. The data management method according to claim 2, wherein generating a task record file comprises: If the lease has not expired, a task record file is generated; The recording of the operation execution progress in the task record file includes: If the data operation of the current task is completed and the lease has not expired, the progress of the operation execution used to indicate the completion is recorded in the task record file; If the data operation of the current task is completed, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file, including: If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file; Among the multiple data nodes that are communicatively connected to the shared storage layer, only one data node has a lease at the same time, and the data node with the lease is used to execute the current task.

9. The data management method according to claim 2, further comprising: If the lease is obtained for the first time, the coverage ID is obtained, where the coverage ID is greater than the IDs of all started tasks for the shared storage layer; If the ID of the data record file with the smallest difference from the coverage ID is not less than the ID of the task record file with the smallest difference from the coverage ID, a data record file is generated and the coverage ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the coverage ID is recorded in the generated data record file; If the ID of the data record file with the smallest difference with the overwrite ID is smaller than the ID of the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed, a data record file is generated and the generated data record file is marked with the overwrite ID, and index information is recorded in the generated data record file based on the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed.

10. The data management method according to claim 1, further comprising: A data read request is received, a data record file with a largest ID is obtained, and data related to the data read request in the shared storage layer is read based on index information in the data record file with the largest ID.

11. A data management device, comprising: An acquisition module, used to acquire the ID of the current task, wherein the ID of the current task is greater than the IDs of all started tasks for the shared storage layer; An execution module, used to execute the data operation of the current task and mark the ID of the current task for the data uploaded to the shared storage layer during the data operation; The recording module is used to generate a data recording file and mark the ID of the current task for the data recording file if the data operation of the current task is completed, and record index information in the data recording file, wherein the index information is used to index all data in the shared storage layer.

12. The data management device according to claim 11, further comprising a task module, configured to: Generate a task record file and mark the task record file with the ID of the current task; During the data operation process of executing the current task, the operation execution progress and / or the data range of the data uploaded to the shared storage layer during the operation execution process are recorded in the task record file.

13. The data management device according to claim 12, wherein the recording module is used for: If the operation execution progress recorded in the task record file is completed, a data record file is generated and the ID of the current task is marked in the data record file, and index information is recorded in the data record file.

14. The data management device according to claim 11, wherein the acquisition module is used for: Get the ID record file with the largest ID, and increment it based on the ID of the ID record file to get the ID of the current task; Build the ID record file corresponding to the ID of the current task.

15. The data management device according to claim 11, wherein the execution module is used to execute the data operation of the current task, and when marking the ID of the current task for the data uploaded to the shared storage layer during the data operation, it is used to: Upload the data to be uploaded of the current task to the shared storage layer in the form of an object file; For each object file in the at least one object file uploaded to the shared storage layer, the ID of the current task and the upload order of the object file in the current task are used as the name of the object file.

16. The data management device according to claim 12, wherein when the task module is used to generate a task record file, it is used to: If the lease has not expired, a task record file is generated; The task module is used to record the operation execution progress in the task record file, and is used to: If the data operation of the current task is completed and the lease has not expired, the progress of the operation execution used to indicate the completion is recorded in the task record file; The recording module is used to: If the data operation of the current task is completed and the lease has not expired, a data record file is generated and the ID of the current task is marked for the data record file, and index information is recorded in the data record file; in, Among the multiple data nodes that are communicatively connected to the shared storage layer, only one data node has a lease at the same time, and the data node with the lease is used to execute the current task.

17. The data management device according to claim 12, further comprising a re-determination module, configured to: If the lease is obtained for the first time, the coverage ID is obtained, where The overlay ID is greater than the IDs of all started tasks targeting the shared storage layer; If the ID of the data record file with the smallest difference from the coverage ID is not less than the ID of the task record file with the smallest difference from the coverage ID, a data record file is generated and the coverage ID is marked for the generated data record file, and index information in the data record file with the smallest difference from the coverage ID is recorded in the generated data record file; If the ID of the data record file with the smallest difference with the overwrite ID is smaller than the ID of the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed, a data record file is generated and the generated data record file is marked with the overwrite ID, and index information is recorded in the generated data record file based on the task record file with the smallest difference with the overwrite ID and the recorded operation execution progress is completed.

18. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

19. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

20. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images