A data storage method, device and electronic equipment

Abstract

The application provides a data storage method, device and electronic equipment, and belongs to the technical field of data storage. The data storage method of the application comprises the following steps: in response to the metadata corresponding to the data table in the database and maintained independently by the server being changed, setting the second data table to an unmodifiable state, and writing the incremental data in the second data table into a first data table in a persistent storage space on the server; creating a new second data table corresponding to the changed metadata, and recording the newly obtained incremental data into the second data table, so that the changed metadata can take effect in real time without affecting the data writing operation, while greatly reducing the storage space of the metadata.

Description

Technical Field

[0001] This application belongs to the field of data storage technology, specifically relating to a data storage method, apparatus, and electronic device. Background Technology

[0002] The user data stored in the database can include data and metadata. The metadata, referred to as the data's data, is used to interpret the data. If the user data is structured data, such as data recorded in a table structure, then the metadata can include basic information used to interpret the table structure. After defining a table structure, the table structure can be adjusted using predefined modification statements, resulting in changes to the corresponding metadata. When the metadata changes, different versions of metadata need to be used in conjunction with different versions of data: the new version of metadata may not accurately interpret the old version of data; the old version of metadata may not be able to interpret the new version of data.

[0003] Regarding changes to metadata, related technologies can modify the metadata based on the original table structure. However, it is necessary to temporarily stop writing new data and wait for the metadata change to be completed before continuing to write. This affects the write operations to the database and consequently the business. Summary of the Invention

[0004] In view of this, this application provides a data storage method, apparatus, and electronic device to solve the problem that when metadata changes are performed, it is necessary to temporarily stop writing new data, thereby affecting the write operation to the database.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] Firstly, a data storage method is provided, applied to a server, on which a database is deployed; the database includes several first data tables stored in persistent storage space on the server; and second data tables stored in cache space on the server; the second data tables are used to store incremental data to be written to the first data tables; the method includes:

[0007] In response to a change in the metadata corresponding to the data table in the database that is independently maintained by the server, the second data table is set to an immutable state, and the incremental data in the second data table is written into the first data table in the persistent storage space on the server.

[0008] Create a new second data table corresponding to the changed metadata, and record the newly acquired incremental data in the second data table.

[0009] Optionally, the server includes a metadata manager, which is used to maintain metadata corresponding to data tables in the database and identifiers of the metadata; the first data table stores identifiers of metadata corresponding to the first data table; the second data table stores identifiers of metadata corresponding to the second data table.

[0010] Optionally, the metadata manager maintains metadata in the form of key-value pairs; wherein the key in the key-value pair is the identifier of the metadata, and the value in the key-value pair is the metadata.

[0011] Optionally, the database is a database using the LSM-Tree storage engine.

[0012] Optionally, before setting the second data table to an immutable state and writing incremental data from the second data table to the first data table in the persistent storage space on the server in response to a change in the metadata corresponding to the data table independently maintained by the server, the method further includes:

[0013] Determine whether any of the preset change conditions are met; if so, determine that the metadata corresponding to the data table in the database has been changed.

[0014] The change conditions include at least one of the following:

[0015] The metadata maintained in the metadata manager corresponding to the data table in the database has changed;

[0016] The metadata corresponding to the newly acquired incremental data is different from the metadata corresponding to the second data table.

[0017] Optionally, the metadata is used to interpret the data entries written to the data table; the method further includes:

[0018] In response to a data query request, a target data entry is queried from the plurality of first data tables, and metadata corresponding to the target data entry is retrieved from the metadata manager based on the identifier of the metadata corresponding to the first data table where the target data entry is located, so as to interpret the target data entry.

[0019] Optionally, the method further includes:

[0020] In response to a data table merge request for the plurality of first data tables, the plurality of first data tables are merged into a target first data table.

[0021] Optionally, merging the plurality of first data tables into a target first data table includes:

[0022] Based on the identifier of the metadata stored in each of the plurality of first data tables, the latest metadata is obtained from the metadata manager and used as the target metadata;

[0023] According to the target metadata, each data entry in the plurality of first data tables is stored in the corresponding position in the target first data table.

[0024] Secondly, a data storage device is provided, the device comprising:

[0025] The first execution module is configured to, in response to a change in the metadata corresponding to the data table in the database that is independently maintained by the server, set the second data table to an unmodifiable state and write the incremental data in the second data table into the first data table in the persistent storage space on the server.

[0026] The second execution module is used to create a new second data table corresponding to the changed metadata and record the newly acquired incremental data into the second data table.

[0027] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0028] Fourthly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0029] By independently maintaining the metadata corresponding to the data tables in the database on the server side, instead of saving the metadata to the data tables, when the metadata changes, it is not necessary to update the metadata in the data tables. Instead, a new second data table corresponding to the changed metadata is created to store the incremental data obtained after the metadata change. This greatly reduces the storage space of the metadata, and the changed metadata can take effect in real time without affecting the data writing operation. Attached Figure Description

[0030] Figure 1 This is a schematic flowchart illustrating a data storage method according to an exemplary embodiment of this application;

[0031] Figure 2 This is a schematic diagram of data storage for a server-side implementation of a data storage method, as shown in an exemplary embodiment of this application.

[0032] Figure 3This is a schematic diagram of data storage for a server-side implementation of a data storage method, as shown in an exemplary embodiment of this application.

[0033] Figure 4 This is a flowchart illustrating a first data table merging operation according to an exemplary embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the structure of a data storage device shown in an exemplary embodiment of this application;

[0035] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0039] The user data stored in the database can include data and metadata, where the metadata, referred to as data of the data, is used to interpret the data. For user data of structured data types, such as user data recorded in a table structure, the metadata can include metadata used to interpret the basic information of the table structure, also referred to as table metadata. The table metadata can include metadata used to interpret the following two types of information:

[0040] Basic information about the table: compression algorithm, encryption algorithm, data block size used for storage, number of attributes, etc.;

[0041] Basic information about attributes, also known as basic information about columns: the type of each column, the column index, the default value of the column, etc.

[0042] It should be noted that the changes to metadata described in this specification mainly refer to changes to metadata related to attributes.

[0043] The data populated in the table structure defined by the metadata is the data in the user data. The data cannot exist independently and needs to be interpreted based on the corresponding metadata.

[0044] After defining a table structure, adjustments can be made to the table structure using predefined modification statements, which is called performing a metadata change on the table structure. For example, adding attributes, deleting attributes, and modifying attribute types are called changes to the basic information of the attributes. Metadata changes result in a new table structure. When metadata changes, different versions of metadata need to be used in conjunction with different versions of data, which can easily cause the following problems:

[0045] The new version of metadata may not be able to accurately interpret the old version of data. For example, after performing an operation to delete an attribute and then an operation to add an attribute on the table structure, if the old version of data is not rewritten in time when the attribute is deleted and added, the new version of metadata may be used to interpret the old version of data, which may result in incorrect interpretation of the attributes.

[0046] The old version of metadata cannot interpret the new version of data. For example, if an operation of adding an attribute is performed on the table structure, the old version of metadata cannot interpret the new version of data because it does not contain basic information about the added attribute.

[0047] Regarding changes to metadata, the relevant technologies mainly employ the following two methods:

[0048] 1. Copy Method: This method creates a new table structure that is not yet visible to the user, using the new version of the metadata. Data from the original table structure is queried and rewritten into the new table structure. After the data writing is complete, the new table structure replaces the original table structure, and the original table structure is deleted. The new version of the metadata replaces the old version. The disadvantage of this approach is that for tables with large amounts of data, data querying and writing can consume a significant amount of time. Before the data writing is complete, data exists simultaneously in both the original and new table structures, resulting in multiple copies of data and increasing storage space requirements.

[0049] 2. Replacement (INPLACE) method: Metadata changes are made based on the original table structure. Since the data table used to store the data also stores the corresponding metadata, it is necessary to temporarily stop writing new data when making metadata changes, and continue writing after the metadata changes are completed. This affects the write operations to the database, and thus affects business operations.

[0050] In view of this, this specification proposes a data storage method, which deploys a database on a server; the database includes several first data tables stored in a persistent storage space on the server; and a second data table stored in a cache space on the server; the second data table is used to store incremental data to be written to the first data table; and metadata corresponding to the data tables in the database is also independently maintained on the server.

[0051] In implementation, in response to a change in the metadata corresponding to the data table in the database that is independently maintained by the server, the second data table is set to an immutable state, and the incremental data in the second data table is written to the first data table in the persistent storage space on the server; a new second data table corresponding to the changed metadata is created, and the newly acquired incremental data is recorded in the second data table.

[0052] In the above technical solution, the metadata corresponding to the data table in the database is maintained independently by the server, instead of being saved to the data table. When the metadata changes, it is no longer necessary to update the metadata in the data table. Instead, a new second data table corresponding to the changed metadata is created to store the incremental data obtained after the metadata change. This greatly reduces the storage space of the metadata, and the changed metadata can take effect in real time without affecting the data writing operation.

[0053] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0054] Please see Figure 1 , Figure 1 This application provides a data storage method. The method is applied to a server, on which a database is deployed. The database includes several first data tables stored in persistent storage space on the server, and a second data table stored in cache space on the server. The second data table stores incremental data to be written into the first data table.

[0055] The server can be a server or server cluster used to provide services to users, and no specific limitation is made here.

[0056] The database deployed on the server can be diverse, such as a key-value store database, a column-oriented database, a document-oriented database, a graph database, and so on. In one embodiment, the database deployed on the server in this specification is a database using a Log Structured Merge-Tree (LSM-Tree) storage engine.

[0057] like Figure 2 As shown, the data stored in the database includes two parts: several first data tables stored in the persistent storage space of the server and second data tables stored in the cache space of the server.

[0058] The persistent storage space on the server side can specifically be a disk or hard drive. The first data table in the persistent storage space on the server side can be an internally ordered file, also known as a Sorted String Table (SSTable). The data stored in the first data table can be called baseline data. The baseline data can include several data entries, which can be interpreted by metadata corresponding to the version of the first data table.

[0059] The persistent storage space on the server side can store several first data tables. Each first data table corresponds to a version of metadata. Different first data tables can correspond to different versions of metadata, or they can correspond to the same version of metadata. For example... Figure 2 As shown, the persistent storage space includes a first data table 1, a first data table 2, and a first data table 3, wherein the first data table 1 and the first data table 2 correspond to the same metadata of version A1, and the first data table 3 corresponds to the metadata of version A2.

[0060] The server-side cache space can specifically be the server's memory. Within this cache space, a second data table, also known as a memory table (MemTable), is stored. The data stored in this second data table can be called incremental data. Each incremental data entry can be a newly added data entry, which can be interpreted by the metadata corresponding to the version of the second data table, such as... Figure 2 As shown, the version of the metadata corresponding to the second data table is A2.

[0061] The server also independently maintains metadata corresponding to the database tables. This can include version management of the metadata, which is used to interpret data entries written to the data tables. Different versions of the metadata are used to interpret the data entries written to the corresponding data tables. The server can maintain metadata in various ways, and this specification does not impose specific limitations; for example, as shown in... Figure 2 Taking the metadata manager shown as an example, the server includes a metadata manager, which is used to maintain metadata corresponding to the data tables in the database. After the server receives the user's modification statement for the metadata, the metadata manager modifies the metadata and performs version management.

[0062] In the server-side configuration described above, the steps of the data storage method are as follows.

[0063] S110. In response to a change in the metadata corresponding to the data table in the database that is independently maintained by the server, the second data table is set to an unmodifiable state, and the incremental data in the second data table is written into the first data table in the persistent storage space on the server.

[0064] When the server obtains incremental data, it can write the incremental data to the second data table in the cache. At this time, the second data table is in a modifiable state and can be called a modifiable second data table.

[0065] During the process of acquiring incremental data, the server can monitor whether the metadata corresponding to the data table in the database has changed.

[0066] If the server determines that the metadata corresponding to the data table in the database has not changed, then when the incremental data in the current second data table reaches a preset data volume threshold, the second data table can be set to an immutable state, becoming as follows. Figure 2 The immutable second data table shown is then used to write incremental data from this immutable second data table into the persistent storage space on the server to become baseline data.

[0067] If the server determines that the metadata corresponding to the data table in the database has changed, then regardless of whether the incremental data in the second data table has reached the preset data volume threshold, the second data table will be set to an immutable state, and the incremental data in the immutable second data table will be written to the server's persistent storage space to become the baseline data.

[0068] There are various ways to write incremental data from the second data table to the persistent storage space on the server. For example, the entire second data table can be imported from the cache space into the persistent storage space to become a new first data table, which corresponds to the same version of metadata as the second data table. Alternatively, incremental data from the second data table can be written into a first data table in the persistent storage space that has the same version of metadata as the second data table. In one implementation, it can be first checked whether a first data table with the same version of metadata as the second data table exists in the persistent storage space. If it does not exist, a new first data table with the same version of metadata as the second data table is created in the persistent storage space, and then the incremental data from the second data table is written into the new first data table. If it exists, it can be checked whether the baseline data in the first data table with the same version of metadata reaches a preset threshold. If it does not reach the threshold, the incremental data from the second data table is merged into the first data table with the same version of metadata. If it reaches the threshold, a new first data table with the same version of metadata as the second data table is created in the persistent storage space, and then the incremental data from the second data table is written into the new first data table.

[0069] There are various ways for the server to determine whether the metadata corresponding to the data table in the database has changed. It can preset multiple change conditions and determine whether the metadata corresponding to the data table in the database has changed by checking whether any of the preset change conditions is met; if the condition is met, then it is determined that the metadata corresponding to the data table in the database has changed.

[0070] The change conditions can be set according to actual needs and may include at least one of the following:

[0071] The metadata maintained in the metadata manager that corresponds to the data table in the database changes; for example, the server may obtain a user's modification statement for the metadata; or, obtain a modification statement for the attribute of the metadata; or, the metadata manager may obtain a message indicating that the metadata has changed after completing the modification of the metadata, and the message indicating that the metadata has changed may include the identifier of the changed metadata, etc.

[0072] The metadata corresponding to the newly acquired incremental data is different from the metadata corresponding to the second data table; for example, the version of the metadata corresponding to the newly acquired incremental data is different from the version of the metadata corresponding to the second data table; or, the identifier of the metadata corresponding to the newly acquired incremental data is different from the identifier of the metadata stored in the second data table; or, the basic information of any attribute in the metadata corresponding to the newly acquired incremental data is different from the basic information of any attribute in the metadata corresponding to the second data table; or, the number of attributes corresponding to the data entries included in the newly acquired incremental data is different from the number of attributes corresponding to the data entries in the incremental data already stored in the second data table, etc.

[0073] It should be noted that the server determines whether the metadata corresponding to the data table in the database has changed. This can be for any single change to the metadata; or it can be for changes only to the metadata related to attributes; or it can be for changes only to the metadata related to certain information about the attributes. These certain information can be set according to actual needs, such as the number of attributes, the order of attributes, etc. If only the metadata related to attributes has changed, after determining that the metadata corresponding to the data table in the database has changed, the server also needs to determine whether this change is for metadata related to attributes. If so, the second data table is set to an immutable state, and the incremental data in the second data table is written to the first data table in the persistent storage space on the server. If not, it still proceeds as if the metadata has not changed.

[0074] S120. Create a new second data table corresponding to the changed metadata, and record the newly acquired incremental data in the second data table.

[0075] On the server side, while setting the second data table to an immutable state and writing incremental data from the second data table into the first data table in the persistent storage space on the server, the server simultaneously creates a new second data table in the cache space. This new second data table corresponds to the modified metadata; that is, the new second data table corresponds to the version of the modified metadata. For example, when... Figure 2 After the incremental data in the second data table shown is written to the first data table in the persistent storage space on the server, the version of the metadata corresponding to the new second data table created by the server in the cache space can be A3 or A1.

[0076] To facilitate metadata management, the metadata manager configures corresponding identifiers for different versions of metadata. The metadata manager maintains the metadata corresponding to the data tables in the database and the corresponding identifiers for each metadata element. The identifier configured by the metadata manager for each metadata element can be a unique identifier, specifically derived from the version number of each metadata element; for example, it can be formed by concatenating the table's unique identifier and the metadata's version number.

[0077] When the server receives a user's modification statement for metadata, the metadata manager can modify the metadata and determine whether a new identifier needs to be configured.

[0078] It should be noted that the metadata manager can also provide concurrent locks to prevent concurrent modification problems caused by multiple threads operating on the same key-value pair; in addition, frequently accessed key-value pairs can be cached in the cache space to reduce disk read and write caused by metadata access.

[0079] Accordingly, to indicate the correspondence between each data table and its metadata, the first data table may store an identifier for the metadata corresponding to the first data table; the second data table may store an identifier for the metadata corresponding to the second data table. For example... Figure 3 As shown, identifier aA1 is stored in the first data table 1 and the second data table 2 to indicate the metadata of the first data table 1 and the first data table 2 corresponding to version A1; identifier aA2 is stored in the first data table 3 to indicate the metadata of the first data table 3 corresponding to version A2; identifier aA2 is stored in the second data table to indicate the metadata of the second data table corresponding to version A2.

[0080] The server can retrieve the metadata corresponding to each data table from the metadata manager based on the identifier of the metadata stored in each data table.

[0081] The metadata manager can maintain metadata corresponding to the data tables in the database and the identifiers of that metadata in various ways, and this specification does not impose specific limitations. For example, metadata can be maintained in the form of key-value pairs, that is, a key-value storage engine can be used to store metadata. The key-value pair can be represented as "metadata identifier, metadata"; wherein the key in the key-value pair is the identifier of the metadata, and the value in the key-value pair is the metadata; alternatively, the metadata manager can maintain a table that establishes a correspondence between each piece of metadata and its identifier. Figure 3As shown, the key-value pairs maintained by the metadata manager can include "key = aA1, value = metadata B1" and "key = aA2, value = metadata B2". Then, the server can query the metadata manager to find the metadata B1 corresponding to aA1, i.e., the metadata of version A1, based on the identifier aA1 stored in the data table; and query the metadata manager to find the metadata B2 corresponding to aA2, i.e., the metadata of version A2, based on the identifier aA2 stored in the data table.

[0082] This application embodiment maintains the metadata corresponding to the data tables in the database independently on the server side, instead of saving the metadata to the data tables. When the metadata changes, it is not necessary to pause the write operation to update the metadata in the data tables. Instead, a new second data table corresponding to the changed metadata is created to store the incremental data obtained after the metadata change. This greatly reduces the storage space of the metadata, and the changed metadata can take effect in real time without affecting the data write operation.

[0083] Based on the above embodiments, when it is necessary to query data from the database, a data query request can be sent to the server. The method further includes:

[0084] Upon receiving a data query request, the server can check whether a target data entry corresponding to the query request exists in the plurality of first data tables. If it exists, the server reads the target data entry and the identifier of the metadata corresponding to the first data table from the first data table containing the target data entry, and retrieves the metadata corresponding to the target data entry from the metadata manager based on the identifier of the metadata corresponding to the first data table containing the target data entry, in order to interpret the target data entry. The server then replies to the user with the query results obtained based on the target data entry and the corresponding metadata, which may be, for example, the table structure populated with the target data entry.

[0085] It should be noted that, in response to receiving a data query request, before querying the plurality of first data tables to see if a target data entry corresponding to the data query request exists, the server may first query the second data table to see if a target data entry corresponding to the data query request exists. If it does not exist, the server then queries the plurality of first data tables to see if a target data entry corresponding to the data query request exists. If it exists, the server reads the target data entry and the identifier of the metadata corresponding to the second data table from the second data table, and queries the metadata manager to find the metadata corresponding to the target data entry based on the identifier of the metadata corresponding to the second data table, so as to interpret the target data entry.

[0086] The metadata manager can significantly reduce the storage space of metadata and accurately retrieve the metadata of each version corresponding to the data entry, thereby enabling accurate interpretation of each data entry.

[0087] Based on the above embodiments, in order to reduce the storage space requirements caused by the ever-increasing number of first data tables and the amount of data they store in the persistent storage space, a merging operation can be performed periodically on several first data tables in the persistent storage space. In response to a data table merging request for the several first data tables, the server merges the several first data tables into a target first data table. The merging operation process can be as follows.

[0088] The server determines several first data tables that need to be merged based on the data table merge request. The first data tables that need to be merged can be some of the first data tables specified in all the first data tables in the persistent storage space, or they can be several newly generated first data tables in the persistent storage space.

[0089] After identifying the first data tables that need to be merged, the server iterates through all the first data tables, reads the identifiers of the metadata stored in each first data table, and then retrieves the latest metadata from the metadata manager based on the identifiers of the metadata stored in each first data table, using it as the target metadata. The latest metadata can refer to the latest metadata among the metadata corresponding to the identifiers of the metadata stored in each first data table, or it can refer to the latest metadata among all versions of metadata in the metadata manager; for example, ... Figure 4 As shown, the first data table requiring a merge operation includes first data table 1, first data table 2, and first data table 3; wherein, the metadata identifier stored in first data table 1 and second data table 2 is aA1; and the metadata identifier stored in first data table 3 is aA2. The metadata corresponding to aA1 is found to be version A1 metadata from the metadata manager, and the metadata corresponding to aA2 is version A2 metadata. If the latest metadata in the metadata manager is version A2 metadata, then version A2 metadata is used as the target metadata, and the identifier of the metadata stored in the generated target first data table is aA2, as shown below. Figure 4 As shown; if the latest metadata in the metadata manager is the metadata of version A3, then the metadata of version A3 is used as the target metadata, and the identifier of the metadata stored in the generated target first data table is aA3.

[0090] After the target metadata is determined on the server side, each data entry in the plurality of first data tables can be stored in the corresponding location in the target first data table according to the target metadata. For example... Figure 4As shown, the server can first read the data stored in each first data table according to the metadata corresponding to each first data table. The data stored in each first data table includes several data entries. Then, the data in the data entries read from each first data table is projected according to the target metadata and adjusted to conform to the target metadata, that is, adjusted to be data that can be interpreted by the target metadata. After the data adjustment is completed, the adjusted data is written into the generated target first data table according to the preset sorting rules, and the identifier corresponding to the target metadata is stored in the target first data table.

[0091] Specifically, the data read from each of the first data tables will be adjusted according to the target metadata. Specific adjustment methods may include:

[0092] If any attribute that exists in the metadata of the first data table does not exist in the target metadata, the data corresponding to that attribute in each data entry read from the first data table will be discarded.

[0093] If any attribute in the target metadata does not exist in the metadata corresponding to the first data table, then add the position of the data corresponding to that attribute to each data entry read from the first data table and fill it with the default value corresponding to that attribute.

[0094] If the metadata corresponding to the first data table is different from the basic information of any attribute in the target metadata, then the data corresponding to any attribute in each data entry read from the first data table will be adjusted according to the basic information of that attribute in the target metadata.

[0095] It should be noted that as metadata changes continuously, the amount of metadata stored in the metadata manager will also increase. Therefore, the metadata manager can provide a periodic metadata reclamation mechanism to periodically clean up metadata. For example, the decision to clean up can be determined based on the correspondence between each piece of metadata and the first data table, and the lifespan of the metadata. The identifiers of metadata stored in all the first data tables in the persistent storage space can be periodically traversed, and metadata corresponding to identifiers of metadata that do not exist in the metadata manager during the traversal can be cleaned up. Metadata requiring cleanup can also be required to meet the condition that its existence time has exceeded a preset lifespan.

[0096] By merging several first data tables in the persistent storage space, some storage space can be freed up, reducing the storage pressure on the persistent storage space. Furthermore, the data in the several first data tables can be adjusted to conform to the latest metadata through the merging operation, thereby achieving data integration.

[0097] It's important to note that a single table structure can be divided into multiple partitions, each corresponding to its own first data table, second data table, and metadata. Depending on the partitioning method, different partitions can correspond to different metadata or the same metadata. If different partitions correspond to the same metadata, the same key-value pairs can be used in the metadata manager, significantly reducing metadata redundancy and storage requirements.

[0098] Corresponding to the embodiments of the aforementioned data storage method, this application also provides embodiments of a data storage device.

[0099] like Figure 5 As shown, the data storage device includes: a first execution module 501 and a second execution module 502.

[0100] The first execution module 501 is used to respond to changes in the metadata corresponding to the data table in the database that is independently maintained by the server, set the second data table to an unmodifiable state, and write the incremental data in the second data table into the first data table in the persistent storage space on the server; the second execution module 502 is used to create a new second data table corresponding to the changed metadata, and record the newly acquired incremental data into the second data table.

[0101] Optionally, the server includes a metadata manager, which is used to maintain metadata corresponding to data tables in the database and identifiers of the metadata; the first data table stores identifiers of metadata corresponding to the first data table; the second data table stores identifiers of metadata corresponding to the second data table.

[0102] Optionally, the metadata manager maintains metadata in the form of key-value pairs; wherein the key in the key-value pair is the identifier of the metadata, and the value in the key-value pair is the metadata.

[0103] Optionally, the database is a database using the LSM-Tree storage engine.

[0104] Optionally, the first execution module 501 is further configured to:

[0105] Determine whether any of the preset change conditions are met; if so, determine that the metadata corresponding to the data table in the database has been changed.

[0106] The change conditions include at least one of the following:

[0107] The metadata maintained in the metadata manager corresponding to the data table in the database has changed;

[0108] The metadata corresponding to the newly acquired incremental data is different from the metadata corresponding to the second data table.

[0109] This application embodiment maintains the metadata corresponding to the data tables in the database independently on the server side, instead of saving the metadata to the data tables. When the metadata changes, it is not necessary to pause the write operation to update the metadata in the data tables. Instead, a new second data table corresponding to the changed metadata is created to store the incremental data obtained after the metadata change. This greatly reduces the storage space of the metadata, and the changed metadata can take effect in real time without affecting the data write operation.

[0110] Optionally, the metadata is used to interpret the data entries written to the data table; the first execution module 501 is further used to:

[0111] In response to a data query request, a target data entry is queried from the plurality of first data tables, and metadata corresponding to the target data entry is retrieved from the metadata manager based on the identifier of the metadata corresponding to the first data table where the target data entry is located, so as to interpret the target data entry.

[0112] The metadata manager can significantly reduce the storage space of metadata and accurately retrieve the metadata of each version corresponding to the data entry, thereby enabling accurate interpretation of each data entry.

[0113] Optionally, the first execution module 501 is further configured to: in response to a data table merging request for the plurality of first data tables, merge the plurality of first data tables into a target first data table.

[0114] Optionally, the first execution module 501 is further configured to:

[0115] Based on the identifier of the metadata stored in each of the plurality of first data tables, the latest metadata is obtained from the metadata manager and used as the target metadata;

[0116] According to the target metadata, each data entry in the plurality of first data tables is stored in the corresponding position in the target first data table.

[0117] By merging several first data tables in the persistent storage space, some storage space can be freed up, reducing the storage pressure on the persistent storage space. Furthermore, the data in the several first data tables can be adjusted to conform to the latest metadata through the merging operation, thereby achieving data integration.

[0118] The data storage device embodiments of this application can be applied to electronic devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the electronic device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of an electronic device containing the data storage device of this application. Except for... Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device is located in the embodiment may also include other hardware depending on the actual function of the electronic device, which will not be described in detail here.

[0119] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0120] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0121] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the data storage method described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0123] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0124] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0125] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0126] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0127] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0128] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0129] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0130] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data storage method, characterized in that, The system is applied to a server-side application, on which a database is deployed; the database includes several first data tables stored in a persistent storage space on the server-side application; and second data tables stored in a cache space on the server-side application. The second data table is used to store incremental data to be written into the first data table; The server includes a metadata manager, which is used to maintain metadata corresponding to the data tables in the database and the identifiers of the metadata; The first data table stores identifiers of metadata corresponding to the first data table; The second data table stores identifiers for metadata corresponding to the second data table; The method includes: In response to a change in the metadata corresponding to the data table in the database that is independently maintained by the server, the second data table is set to an immutable state, and the incremental data in the second data table is written into the first data table in the persistent storage space on the server. Create a new second data table corresponding to the changed metadata, and record the newly acquired incremental data in the second data table.

2. The method according to claim 1, characterized in that, The metadata manager maintains metadata in the form of key-value pairs; wherein the key in the key-value pair is the identifier of the metadata, and the value in the key-value pair is the metadata.

3. The method according to claim 1, characterized in that, The database is a database that uses the LSM-Tree storage engine.

4. The method according to claim 1, characterized in that, Before setting the second data table to an immutable state and writing incremental data from the second data table to the first data table in the persistent storage space on the server in response to a change in the metadata corresponding to the data table independently maintained by the server, the method further includes: Determine whether any of the preset change conditions are met; if so, determine that the metadata corresponding to the data table in the database has been changed. The change conditions include at least one of the following: The metadata maintained in the metadata manager corresponding to the data table in the database has changed; The metadata corresponding to the newly acquired incremental data is different from the metadata corresponding to the second data table.

5. The method according to claim 1, characterized in that, The metadata is used to interpret the data entries written to the data table; the method further includes: In response to a data query request, a target data entry is queried from the plurality of first data tables, and metadata corresponding to the target data entry is retrieved from the metadata manager based on the identifier of the metadata corresponding to the first data table where the target data entry is located, so as to interpret the target data entry.

6. The method according to claim 1, characterized in that, The method further includes: In response to a data table merge request for the plurality of first data tables, the plurality of first data tables are merged into a target first data table.

7. The method according to claim 6, characterized in that, The step of merging the plurality of first data tables into a target first data table includes: Based on the identifier of the metadata stored in each of the plurality of first data tables, the latest metadata is obtained from the metadata manager and used as the target metadata; According to the target metadata, each data entry in the plurality of first data tables is stored in the corresponding position in the target first data table.

8. A data storage device, characterized in that, The system is applied to a server-side application, on which a database is deployed; the database includes several first data tables stored in a persistent storage space on the server-side application; and second data tables stored in a cache space on the server-side application. The second data table is used to store incremental data to be written into the first data table; The server includes a metadata manager, which is used to maintain metadata corresponding to the data tables in the database and the identifiers of the metadata; The first data table stores identifiers of metadata corresponding to the first data table; The second data table stores identifiers for metadata corresponding to the second data table; The device includes: The first execution module is used to respond to changes in the metadata corresponding to the data table in the database that is independently maintained by the server, set the second data table to an immutable state, and write the incremental data in the second data table into the first data table in the persistent storage space on the server. The second execution module is used to create a new second data table corresponding to the changed metadata and record the newly acquired incremental data into the second data table.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-7.

Images