Data processing method, device, storage medium and computer program product

Abstract

The embodiments of the present application provide a data processing method, device, storage medium and computer program product. Among them, the KV entry introduces a minor version number based on the original KV entry. Based on the newly added minor version number, when the user data is migrated, the newly created KV entry can change the minor version number and keep the major version number unchanged. For user data updates, the newly created KV entry can change the major version number. In this way, even if there is an update to the user data during the user data migration process, the key-value database can accurately distinguish the KV entries before and after the user data update based on the major version number, and can abandon the KV entries before the data update, and retain the KV entries corresponding to the updated user data, so that the data stored in the key-value database can be consistent with the user-side data. When querying data, the KV entries corresponding to the updated data can be used to obtain the updated user data, which helps to improve the accuracy of data query.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, device, storage medium and computer program product. Background Art

[0002] A key-value (KV) database is a non-relational database that uses a simple key-value approach to store data. A key-value database stores data as a collection of key-value pairs, where the key serves as a unique identifier. Both the key and the value can be anything from a simple object to a complex composite object.

[0003] In a key-value database, a global, increasing serial number is assigned to each key-value entry. This serial number is also called the major version number, which is used to distinguish KV entries with the same key. In an existing key-value database, when data is migrated, the key-value database will create a new KV entry with an increasing version number, and expire and abandon the KV entry with the previous lower major version number. When user data is updated, the key-value database will also create a new KV entry with an increasing version number, and expire and abandon the KV entry with the previous lower major version number.

[0004] However, if the user updates the KV entry during the data migration process corresponding to the KV entry, the version number of the KV entry after the original KV entry data migration may be higher than the version number of the KV entry when the data corresponding to the original KV entry is updated, causing the KV entry corresponding to the data update to be abandoned, and further causing the data stored in the key-value database to be inconsistent with the user-side data. Summary of the invention

[0005] Multiple aspects of the present application provide a data processing method, device, storage medium and computer program product to reduce the probability of inconsistency between data stored in a key-value database and user-side data, thereby helping to improve the accuracy of data query.

[0006] The present application provides a data processing method, including:

[0007] Determine the data to be migrated;

[0008] Migrate the data to be migrated from the original storage medium to the target storage medium;

[0009] Determine, from the key-value database, a first key-value entry corresponding to the data to be migrated of the original storage medium;

[0010] Generate a second version number based on the first version number included in the first key-value entry;

[0011] A second key-value entry corresponding to the data to be migrated of the target storage medium is generated using the key contained in the first key-value entry as the key, the first major version number contained in the first key-value entry as the major version number, and the second major version number as the minor version number.

[0012] The embodiment of the present application further provides a computing device, comprising: a memory and a processor; wherein the memory is used to store a computer program;

[0013] The processor is coupled to the memory and is configured to execute the computer program to perform the steps in the above data processing method.

[0014] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned data processing method.

[0015] An embodiment of the present application also provides a computer program product, including: a computer program; when the computer program is executed by a processor, it causes the processor to execute the steps in the above-mentioned data processing method.

[0016] In an embodiment of the present application, a minor version number is introduced into the KV entry based on the original KV entry, and each KV entry includes a major version number and a minor version number. Based on the newly added minor version number, when the user data is migrated, the newly created KV entry can change the minor version number and keep the major version number unchanged. For user data updates, the newly created KV entry can change the major version number. In this way, even if there is an update to the user data during the user data migration process, the major version number of the user data update can be different from the major version number of the user data migration. The key-value database can accurately distinguish the KV entries before and after the user data update based on the major version number, and abandon the KV entries before the data update when querying the user data, while retaining the KV entries corresponding to the updated user data, so that the data stored in the key-value database can be consistent with the user-side data. When the user provides the key of the user data to query the user data, the KV entry corresponding to the updated user data can be used to obtain the updated user data, which helps to improve the accuracy of data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0019] Figure 2 and Figure 3 A flowchart of a data updating method provided in an embodiment of the present application;

[0020] Figure 4 A flowchart of a data query method provided in an embodiment of the present application;

[0021] Figure 5 A flowchart of a method for determining a KV entry snapshot provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0024] In a key-value database, each KV entry is assigned a global, increasing serial number. This serial number is also called the major version number, which is used to distinguish KV entries with the same key. Typically, a KV entry may include the following parts:

[0025] |Key|Type|MajorVersionNumber|UserData|.

[0026] Among them, user data is the value corresponding to the key. Type is used to identify some type information of the KV entry. For example, in some embodiments, the type can identify the usage type of the KV entry, such as marking whether the KV entry is deleted. For a deleted KV entry, the type can be a tombstone identifier.

[0027] In an existing key-value database, when migrating data, a new KV entry with an increased major version number is created, and the KV entry with a lower major version number is expired and abandoned. For example, suppose a KV entry is as follows:

[0028] |data:inode=3|type|major ver=100|data location 1|. (Item 1.1)

[0029] In this KV entry, the key is "inode=3"; the major version number is 100, and the value is "data location 1", that is, "data location 1".

[0030] If the user data stored in data location 1 is migrated to data location 2, a new KV entry is created in the key-value database:

[0031] |data:inode=3|type|major ver=101|data location 2|. (Item 1.2)

[0032] Since the user data has been migrated, the KV entry with the major version number of 100 will be deprecated. When the user queries, the key-value database will obtain the user data corresponding to data location 2 based on the KV entry with the major version number of 101.

[0033] For a key-value database, when user data is updated, the key-value database will also create a new KV entry with an increased version number, and expire and abandon the KV entry with the previous lower major version number. For example, for the above KV entry 1.1, the user data corresponding to data position 1 is updated, and a new KV entry is created:

[0034] |data:inode=3|type|major ver=101|data location 1|. (Item 1.3).

[0035] However, if the user updates the user data of the KV entry during the data migration process corresponding to the KV entry, the data migration may be completed later than the user data update, resulting in the version number of the KV entry after the data migration being higher than the version number of the user data update of the KV entry. In this way, if the updated user data is the migrated data, when the user queries the user data corresponding to the key of the KV entry, the key-value database will query the KV entry after the data migration to obtain the user data before the update.

[0036] For example, for the above KV entry 1.1, if during the data migration process corresponding to the KV entry 1.1, the user updates the user data of the KV entry 1.1, and the data migration is completed later than the user data update, the KV entry after the user data update of the KV entry 1.1 is:

[0037] |data:inode=3|type|major ver=101|data location 1|. (Item 1.4)

[0038] The KV entry after data migration of KV entry 1.1 is:

[0039] |data:inode=3|type|major ver=102|data location 2|. (Item 1.5)

[0040] Since the major version number of KV entry 1.5 is greater than the major version number of KV entry 1.4, the key-value database will abandon KV entry 1.4. If the updated user data has been migrated to data location 2, the user data corresponding to data location 2 in KV entry 1.5 is the data before the user data is updated. This situation causes the user data corresponding to the key value "inode=3" stored in the key-value database to be inconsistent with the user-side data.

[0041] When a user queries the user data corresponding to the key "inode=3", the key-value database will query the user data corresponding to data position 2 in KV entry 1.5. If the updated user data has been migrated to data position 2, the user data corresponding to data position 2 in KV entry 1.5 is the data before the user data is updated, which affects the accuracy of data query.

[0042] In order to solve the above problems, in some embodiments of the present application, a minor version number is introduced into the KV entry based on the original KV entry, and each KV entry includes a major version number and a minor version number. Based on the newly added minor version number, when the user data is migrated, the newly created KV entry can change the minor version number and keep the major version number unchanged. For user data updates, the newly created KV entry can change the major version number. In this way, even if there is an update to the user data during the user data migration process, the major version number of the user data update can be different from the major version number of the user data migration. The key-value database can accurately distinguish the KV entries before and after the user data update based on the major version number, and when querying the user data, the KV entries before the data update are discarded, while the KV entries corresponding to the updated user data are retained, so that the data stored in the key-value database is consistent with the user-side data. When the user provides the key of the user data to query the user data, the KV entry corresponding to the updated user data can be used to obtain the updated user data, which helps to improve the accuracy of data query.

[0043] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0044] It should be noted that the same reference numerals denote the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0045] Figure 1 The following is a flow chart of a data processing method provided in an embodiment of the present application. Figure 1 As shown, the data processing method includes:

[0046] 101. Determine the data to be migrated.

[0047] 102. Migrate the data to be migrated from the original storage medium to the target storage medium.

[0048] 103. Determine, from the key-value database, a first key-value entry corresponding to the data to be migrated of the original storage medium.

[0049] 104. Generate a second version number based on the first version number included in the first key-value entry.

[0050] 105. Generate a second key-value entry corresponding to the to-be-migrated data of the target storage medium using the key included in the first key-value entry as the key, the first major version number included in the first key-value entry as the major version number, and the second major version number as the minor version number.

[0051] In the embodiment of the present application, a minor version number is introduced in the KV entry, and the KV entry may include the following content:

[0052] |Key|Type|Major version number|Minor version number|User data|.

[0053] The minor version number can be implemented as an automatically increasing sequence. The minor version number is attached to the key-value pair together with the major version number to form a KV entry.

[0054] Based on the newly introduced KV entries described above, the embodiments of the present application propose a new key-value data processing method, which will be described below in conjunction with specific embodiments.

[0055] In a key-value database, there may be situations where data migration is required. For example, when a disk storing user data fails, the user data needs to be migrated from the original disk to another disk. For another example, in a disk defragmentation scenario, the fragmented data can be migrated to a certain disk for centralized storage. Therefore, when user data migration is required, the data A to be migrated can be determined in step 101. The data A to be migrated can be the user data corresponding to the key-value pair.

[0056] In some embodiments, the status information of the storage space may be monitored; and when it is detected that the status of the storage space is abnormal, the data migration process is started, and the data stored in the storage space with the abnormal status is used as the data A to be migrated.

[0057] In other embodiments, the storage space utilization of the disk may be obtained; and based on the storage space utilization of the disk, the disk requiring data migration may be determined; and the data in the disk requiring data migration may be determined to be data A to be migrated, and so on.

[0058] The method for determining the data to be migrated shown in the above embodiment is only an example and does not constitute a limitation.

[0059] Further, in step 102, the data A to be migrated may be migrated from the original storage space to the target storage space. In the embodiment of the present application, the method for determining the target storage space is not limited. In some embodiments, based on the amount of data A to be migrated, a storage medium whose free storage space can accommodate the data A to be migrated may be selected; and one storage medium may be selected from the free storage space of the storage medium whose free storage space can accommodate the data A to be migrated as the target storage medium.

[0060] Optionally, in order to reduce storage space fragmentation, a storage medium with the highest storage space utilization rate may be obtained from storage media whose free storage space can accommodate the data A to be migrated, and used as the target storage medium, and so on.

[0061] After determining the target storage medium, the data A to be migrated can be migrated from the original storage medium to the target storage medium. Since the storage location of the data to be migrated has changed, the original KV entry corresponding to the data to be migrated cannot index the migrated data A. Therefore, a new KV entry can be created for the data A to be migrated to the target storage medium.

[0062] Based on the minor version number introduced by the KV entry, during data migration, the newly created KV entry can change the minor version number and keep the major version number unchanged. For data updates, the newly created KV entry can change the major version number. Based on this, when creating a new KV entry for the data A to be migrated to the target storage medium, in step 103, the KV entry corresponding to the data A to be migrated on the original storage medium can be determined from the key-value database.

[0063] In some embodiments, the value in the KV entry may be user data. For embodiments in which the value in the KV entry is user data, step 103 may be implemented as: matching the data A to be migrated in the KV entry of the key-value database to obtain the KV entry containing the data A to be migrated. Further, the KV entry with the largest major version number may be selected from the KV entries containing the data A to be migrated; and the KV entry with the largest minor version number may be selected from the KV entries with the largest major version number as the KV entry corresponding to the data A to be migrated of the original storage medium.

[0064] In other embodiments, since the size of user data may be as small as a few bytes or as large as TB, for user data with a small amount of data, it can be directly stored as the value of the KV entry. For user data with a large amount of data, the KV database cannot effectively process the KV entry with a large amount of data. Therefore, in another embodiment, the value in the KV entry can store the location information of the user data, and track the location information of the data in the KV database.

[0065] In this embodiment, an optional implementation method of step 103 is: using the address information of the original storage medium of the data A to be migrated, matching in the KV entries of the key-value database to obtain the KV entry containing the address information of the original storage medium. Further, the KV entry with the largest major version number can be selected from the KV entries containing the address information of the original storage medium; and the KV entry with the largest minor version number can be selected from the KV entries with the largest major version number as the KV entry corresponding to the data A to be migrated of the original storage medium.

[0066] After determining the KV entry corresponding to the data A to be migrated of the original storage medium, another sub-version number may be generated based on the sub-version number contained in the KV entry corresponding to the data A to be migrated of the original storage medium in step 104. In the embodiment of the present application, for the convenience of description and distinction, the KV entry corresponding to the data A to be migrated of the original storage medium may be defined as the first KV entry; the sub-version number contained in the first KV entry may be defined as the first version number V11; and another sub-version number generated based on the first version number may be defined as the second version number V12.

[0067] In some embodiments, the first version number V11 may be increased by a set gradient to obtain the second version number V12. For example, the first version number V11 may be increased by 1 to obtain the second version number V12, and so on.

[0068] Further, in step 105, the key contained in the first KV entry can be used as the key, the major version number contained in the first KV entry can be used as the major version number, and the secondary version number V12 can be used as the secondary version number to generate a KV entry corresponding to the data A to be migrated in the target storage medium.

[0069] For an embodiment in which the data amount of the data A to be migrated is smaller than the set data amount, the key contained in the first KV entry can be used as the key, the major version number contained in the first KV entry can be used as the major version number, the second version number V12 can be used as the minor version number, and the data A to be migrated can be used as the value to generate a KV entry corresponding to the data A to be migrated of the target storage medium.

[0070] Assume that the first KV entry corresponding to the data A to be migrated on the original storage medium is:

[0071] |inode=3|type: normal|major ver=100|minor ver=0|A|.

[0072] Then, the KV entry corresponding to the to-be-migrated data A of the target storage medium can be implemented as:

[0073] |inode=3|type: normal|major ver=100|minor ver=1|A|.

[0074] The normal type means that the KV entry can be used normally and is not marked for deletion.

[0075] For an embodiment in which the data amount of the data A to be migrated is greater than or equal to the set data amount, the key contained in the first KV entry can be used as the key, the major version number contained in the first KV entry can be used as the major version number, the second version number V12 can be used as the minor version number, and the address information of the target storage medium can be used as the value to generate a KV entry corresponding to the data A to be migrated of the target storage medium.

[0076] Assume that the original storage medium is disk D1 and the target storage medium after migration is disk D2. The first KV entry corresponding to the data A to be migrated on the original storage medium is:

[0077] |inode=3|type:normal|major ver=100|minor ver=0|disk D1|.

[0078] Then, the KV entry corresponding to the to-be-migrated data A of the target storage medium can be implemented as:

[0079] |inode=3|type:normal|major ver=100|minor ver=1|disk D2|.

[0080] After generating the KV entry corresponding to the data A to be migrated of the target storage medium, when the user queries the user data corresponding to the key of the KV entry, the user can use the KV entry corresponding to the data A to be migrated of the target storage medium to obtain the user data corresponding to the key to be queried.

[0081] In the key-value database, when user data is updated, a new KV entry is also created. In the embodiment of the present application, when user data is updated, the new KV entry created can change the major version number to distinguish it from the original KV entry. The data update method provided in the embodiment of the present application is exemplarily described below.

[0082] Figure 2 The following is a flow chart of a data updating method provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0083] 201. Get data update request.

[0084] 202. Obtain, from the data update request, a key corresponding to the data to be updated and target data that replaces the data to be updated.

[0085] 203. Determine a target KV entry containing the key of the data to be updated according to the key of the data to be updated.

[0086] 204. Update the to-be-updated data corresponding to the target KV entry to the target data.

[0087] 205. Generate a second major version number based on the first major version number included in the target KV entry.

[0088] 206. Generate a KV entry corresponding to the target data using the key included in the target KV entry as the key, the second major version number as the major version number, and the set minor version number as the minor version number.

[0089] In this embodiment, the data update request may include: the key corresponding to the data to be updated and the target data to replace the data to be updated. Therefore, the key corresponding to the data to be updated and the target data to replace the data to be updated may be obtained from the data update request. Further, the target KV entry containing the key may be determined based on the key of the data to be updated.

[0090] Optionally, the key of the data to be updated can be matched in the KV entries of the key-value database to obtain the KV entry containing the key; and the KV entry with the largest major version number is selected from the KV entries containing the key; then, the KV entry with the largest minor version number can be obtained from the KV entry with the largest major version number as the target KV entry.

[0091] After determining the target KV entry corresponding to the key containing the data to be updated, the data to be updated may be determined according to the target KV entry. For an embodiment in which the value contained in the target KV entry is user data, the user data may be obtained from the target KV entry as the data to be updated.

[0092] For an embodiment in which the value contained in the target KV entry is the storage address of the user data, the storage address of the user data may be obtained from the target KV entry, and the data stored at the storage address may be determined to be the data to be updated.

[0093] Furthermore, the data to be updated can be updated to the target data to realize the user data update. For the user data update, a new KV entry needs to be created. Specifically, another major version number can be generated based on the major version number contained in the target KV entry. In the embodiment of the present application, for the convenience of description and distinction, the major version number contained in the target KV entry is defined as the first major version number; and another major version number generated based on the first major version number is defined as the second major version number.

[0094] Optionally, the first major version number may be increased by a set gradient to obtain a second major version number, for example, the first major version number may be increased by 1 to obtain the second major version number.

[0095] Furthermore, the key contained in the target KV entry can be used as the key, the second major version number as the major version number, and the set minor version number as the minor version number to generate the KV entry corresponding to the target data. The set minor version number can be any value. For example, the set minor version number can be 0, etc.

[0096] For embodiments in which the target data volume is smaller than the set data volume, a KV entry corresponding to the target data can be generated using the key contained in the target KV entry as the key, the second major version number as the major version number, the set minor version number as the minor version number, and the target data as the value.

[0097] For embodiments in which the target data volume is greater than or equal to the set data volume, a KV entry corresponding to the target data can be generated using the key contained in the target KV entry as the key, the second major version number as the major version number, the set minor version number as the minor version number, and the storage address of the target data as the value.

[0098] In some embodiments, the data to be updated may be the data to be migrated A, and the data update process of the data to be migrated A may occur in the process of migrating the data to be migrated from the original storage medium to the target storage medium. Figure 3 As shown, the data update method can be implemented as follows:

[0099] 301. In the process of migrating the data to be migrated from an original storage medium to a target storage medium, obtain a data update request for the data to be migrated.

[0100] 302. Obtain target data that replaces the data to be migrated from the data update request.

[0101] 303. Update the data to be migrated in the original storage medium to target data.

[0102] 304. Generate a second major version number based on the first major version number included in the first KV entry.

[0103] 305. Generate a third KV entry corresponding to the target data using the key included in the first KV entry as the key, the second major version number as the major version number, and the set minor version number as the minor version number.

[0104] In the process of migrating the data A to be migrated from the original storage medium to the target storage medium, the data update request for updating the data to be migrated obtained, the key of the data to be updated included in the data update request is the key in the first KV entry corresponding to the data to be migrated in the original storage medium. Therefore, according to the key of the data to be updated included in the data update request, it can be determined that the data update request is a data update request for updating the data A to be migrated.

[0105] Since all the data to be migrated have not been migrated to the target storage medium, in the above step 105, the KV entry corresponding to the data A to be migrated in the target storage medium has not been created. Therefore, the target KV entry determined in step 203 is the first KV entry. Therefore, in step 303, the data to be migrated in the original storage medium is updated to the target data. For the description of steps 304 and 305, please refer to the relevant contents of steps 205 and 206 above, which will not be repeated here.

[0106] After the data A to be migrated is updated to the target data, for the embodiment that requires data migration of the data A to be migrated, the target data can also be migrated to the target storage medium, and a KV entry of the target data stored in the target storage medium can be created. For the specific implementation of creating the KV entry of the target data stored in the target storage medium, please refer to the above Figure 1 The relevant content will not be repeated here.

[0107] In this embodiment, the KV entry introduces a minor version number based on the original KV entry, and each KV entry includes a major version number and a minor version number. When the user data is updated, the newly created KV entry can change the major version number. When the user data is migrated, the newly created KV entry can change the minor version number and keep the major version number unchanged. In this way, even if there is an update to the user data during the user data migration process, the major version number of the user data update can be different from the major version number of the user data migration. The key-value database can accurately distinguish the KV entries before and after the user data is updated according to the major version number. When the user data is queried, the KV entries before the data update are discarded, and the KV entries corresponding to the updated user data are retained, so that the data stored in the key-value database can be consistent with the user-side data. When the user provides the key of the user data to query the user data, the KV entry corresponding to the updated user data can be used to obtain the updated user data, which helps to improve the accuracy of data query.

[0108] In addition to providing data migration and data update methods, the embodiments of the present application also provide data query methods. Figure 4 Specific embodiments are given for illustrative purposes.

[0109] like Figure 4 As shown, the data query method may include the following main steps:

[0110] 401. Get data query request.

[0111] 402. Get the key to be queried from the data query request.

[0112] 403. Use the key to be queried to match the KV entries in the key-value database to obtain a first target KV entry containing the key to be queried. The number of the first target KV entry can be one or more. Multiple means 2 or more.

[0113] 404. Determine a second target KV entry with the largest major version number from the first target KV entry. The second target KV entry is part or all of the KV entries in the first target KV entry.

[0114] 405. Determine a third target KV entry with the largest minor version number from the second target KV entry. The third target KV entry is a part or all of the second target KV entry.

[0115] 406. According to the third target KV entry, obtain data corresponding to the key to be queried.

[0116] In some embodiments, the value contained in the third target KV entry is user data. Accordingly, step 406 may be implemented as: obtaining the user data contained in the third target KV entry from the third target KV entry as data corresponding to the key to be queried.

[0117] In some other embodiments, the value contained in the third target KV entry is the storage address of the user data. Accordingly, step 406 can be implemented as: obtaining the storage address contained in the third target KV entry from the third target KV entry; and obtaining the user data stored at the storage address as the data corresponding to the key to be queried.

[0118] 407. Determine a query result based on the data corresponding to the key to be queried.

[0119] In the embodiment of the present application, the specific implementation of step 407 is not limited. In some embodiments, the data corresponding to the key to be queried can be encapsulated according to a set protocol format to obtain a query result.

[0120] In other embodiments, an aggregation operation may be performed on the data corresponding to the key to be queried to obtain the query result and the like.

[0121] The key-value database provided in the embodiment of the present application can also support key-value data deletion and compression. In the embodiment of the present application, after the key-value data is deleted, the storage space storing the data will be discontinuous, resulting in a waste of storage space. Accordingly, key-value data compression refers to compressing the discontinuous storage space into a continuous storage space, which can save storage space. The compressed storage space can be used to store other data, which helps to improve the utilization rate of the storage space.

[0122] In order to increase the robustness of the key-value database, the key-value database can support redundant storage and store snapshots of user data. In this way, if the user data update fails, data query services can also be provided based on the snapshots of the user data. In the embodiment of the present application, when deleting and compressing key-value data, the user can specify the version of the KV entry to be retained and stored in the key-value database as a snapshot of the user data. Figure 5 The following is an example of how to customize the KV entry snapshot.

[0123] like Figure 5 As shown, the KV entry snapshot determination method mainly includes the following steps:

[0124] 501. Get the specified reserved major version number corresponding to the specified key.

[0125] 502. Obtain, from the key-value entries containing the specified key, a fourth target KV entry containing a major version number that is less than or equal to the specified reserved major version number.

[0126] 503. Determine a fifth target KV entry with the largest major version number from the fourth target KV entries.

[0127] 504. Determine a sixth target KV entry with the largest minor version number from the fifth target KV entries.

[0128] 505. Persist the sixth target KV entry as a snapshot of the KV entry corresponding to the specified key.

[0129] In this embodiment, the specified key can be any key that needs to be stored in a snapshot. The key can be specified by the user or by the operation and maintenance side of the key-value database, or all keys in the key-value database can be used by default. In this embodiment, the user can customize the major version number of the KV entry snapshot corresponding to the specified key. Accordingly, the user terminal can provide the specified key and the specified reserved version number to the key-value database.

[0130] Correspondingly, for the key-value database, in step 501, the specified reserved major version number corresponding to the specified key can be obtained. Then, the KV entry containing the specified key can be obtained from the KV entry of the key-value database. In order to ensure the timeliness of user data, the KV entry as a snapshot of the KV entry is as newer as possible. Since both the major version number and the minor version number can be an increasing sequence, in step 502, the fourth target KV entry containing the major version number less than or equal to the specified reserved major version number is obtained from the KV entry containing the specified key. The fourth target KV entry may be one or more. In the case where there are multiple fourth target KV entries, in step 503, the fifth target KV entry with the largest major version number can be determined from the fourth target KV entry. The fifth target KV entry may be part or all of the KV entries of the fourth target KV entry. Further, in step 504, the sixth target KV entry with the largest minor version number can be determined from the fifth target KV entry. The sixth target KV entry may be part or all of the fifth target KV entry. Then, in step 505, the sixth target KV entry may be persisted as a KV entry snapshot corresponding to the specified key, and KV entry redundancy storage may be performed. In this way, when the user data of the higher version KV entry corresponding to the specified key fails to be updated or data is lost, data query services may be provided based on the KV entry snapshot corresponding to the specified key.

[0131] Specifically, for a key-value database, a data query request may be obtained; the data query request may include: a key to be queried. If the key to be queried is a specified key, the specified key may be used to match the KV entry in the key-value database to obtain a target KV entry containing the specified key. For an embodiment in which a KV entry snapshot corresponding to a specified key is stored, the target KV entry containing the specified key includes the KV entry snapshot corresponding to the specified key, that is, the sixth target KV entry.

[0132] In an embodiment of the present application, for the convenience of description and distinction, the target KV entry containing the specified key is defined as the seventh target KV entry. The number of KV entries contained in the seventh target KV entry is 1 or more. The seventh target KV entry contains the above-mentioned sixth target KV entry. In some embodiments, the seventh target KV entry may contain a KV entry whose major version number is greater than the major version number of the sixth target KV entry, and the KV entry whose major version number is greater than the major version number of the sixth target KV entry in the seventh target KV entry is defined as the eighth target KV entry. If the data corresponding to the eighth target KV entry is normal, data query services can be provided based on the eighth target KV entry. Specifically, the target KV entry with the largest major version number can be determined from the eighth target KV entry; and the target KV entry with the largest minor version number can be determined from the target KV entry with the largest major version number. Afterwards, the data corresponding to the specified key can be obtained based on the target KV entry with the largest minor version number; and the query result can be determined based on the data corresponding to the specified key. For the specific implementation of obtaining the data corresponding to the specified key and determining the query result based on the data corresponding to the specified key, please refer to the relevant contents of the above steps 406 and 407, which will not be repeated here.

[0133] Accordingly, if the data corresponding to the eighth target KV entry is missing, the data corresponding to the specified key can be obtained according to the sixth target KV entry; and the query result is determined based on the data corresponding to the specified key. For the specific implementation of this part, please refer to the relevant contents of step 406 and step 407 above, which will not be repeated here. Among them, the storage of KV entry snapshots corresponding to the specified key can provide data query services when the KV entry with a higher version number fails, which helps to improve the robustness of the key-value database.

[0134] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 401 and 402 can be device A; for another example, the execution subject of step 401 can be device A, and the execution subject of step 402 can be device B; and so on.

[0135] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 401, 402, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0136] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned data processing method.

[0137] The present application also provides a computer program product, including: a computer program; when the computer program is executed by a processor, the processor is caused to execute the steps in the above data processing method. In the present application, the specific implementation form of the computer program product is not limited. In some embodiments, the computer program product can be implemented as a storage engine of a key-value database, etc.

[0138] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 6 As shown, the computing device includes: a memory 60a and a processor 60b. The memory 60a is used to store computer programs.

[0139] The processor 60b is coupled to the memory 60a, and is used to execute the computer program for: determining the data to be migrated; migrating the data to be migrated from the original storage medium to the target storage medium; determining, from the key-value database, a first key-value entry corresponding to the data to be migrated of the original storage medium; generating a second version number based on the first version number contained in the first key-value entry; using the key contained in the first key-value entry as the key, the first major version number contained in the first key-value entry as the major version number, and the second version number as the minor version number, to generate a second key-value entry corresponding to the data to be migrated of the target storage medium.

[0140] The above-mentioned original storage medium and target storage medium may be the storage medium of the computing device itself, or may be the storage medium of other physical machines.

[0141] Optionally, when the processor 60b generates the second key-value entry corresponding to the data to be migrated of the target storage medium, it is specifically used to: use the key contained in the first key-value entry as the key, the first major version number contained in the first key-value entry as the major version number, the second major version number as the minor version number, and the address information of the target storage medium as the value to generate the second key-value entry.

[0142] Optionally, when generating the second version number, the processor 60b is specifically configured to: increase the first version number by a set gradient to obtain the second version number.

[0143] In some embodiments, the processor 60b is also used to: in the process of migrating the data to be migrated from the original storage medium to the target storage medium, obtain a data update request for the data to be migrated through the communication component 60c; obtain target data to replace the data to be migrated from the data update request; update the data to be migrated in the original storage medium to the target data; generate a second major version number based on the first major version number; use the key contained in the first key value entry as the key, the second major version number as the major version number, and the set minor version number as the minor version number to generate a third key value entry corresponding to the target data.

[0144] In other embodiments, the processor 60b is also used to: obtain a first data query request through the communication component 60c; obtain the key to be queried from the first data query request; use the key to be queried to match the key value entries in the key value database to obtain a first target key value entry containing the key to be queried; determine the second target key value entry with the largest major version number from the first target key value entry; determine the third target key value entry with the largest minor version number from the second target key value entry; obtain the data corresponding to the key to be queried according to the third target key value entry; and determine the query result based on the data corresponding to the key to be queried.

[0145] In some further embodiments, the processor 60b is also used to: obtain a specified reserved major version number corresponding to a specified key; obtain a fourth target key value entry whose major version number is less than or equal to the specified reserved major version number from the key value entry containing the specified key; determine a fifth target key value entry with the largest major version number from the fourth target key value entry; determine a sixth target key value entry with the largest minor version number from the fifth target key value entry; and persist the sixth target key value entry as a snapshot of the key value entry corresponding to the specified key.

[0146] Optionally, the processor 60b is also used to: obtain a second data query request; when the key to be queried contained in the second data query request is the specified key, use the specified key to match in the key value entries to obtain a seventh target key value entry containing the specified key; the seventh target key value entry contains the sixth target key value entry; if the seventh target key value entry contains an eighth target key value entry whose major version number is greater than the sixth target key value entry, when the data corresponding to the eighth target key value entry is missing, obtain the data corresponding to the specified key according to the sixth target key value entry; and determine the query result based on the data corresponding to the specified key.

[0147] In some optional embodiments, such as Figure 6 As shown, the computing device may further include optional components such as a power supply component 60d. Figure 6The components are shown schematically only and do not necessarily include Figure 6 The components shown do not necessarily mean that the computing device can only include Figure 6 Components shown.

[0148] The computing device provided in this embodiment introduces a minor version number into the KV entry based on the original KV entry, and each KV entry includes a major version number and a minor version number. Based on the newly added minor version number, when the user data is migrated, the newly created KV entry can change the minor version number and keep the major version number unchanged. For user data updates, the newly created KV entry can change the major version number. In this way, even if there is an update to the user data during the user data migration process, the major version number of the user data update can be different from the major version number of the user data migration. The key-value database can accurately distinguish the KV entries before and after the user data update based on the major version number, and when querying the user data, the KV entries before the data update are discarded, while the KV entries corresponding to the updated user data are retained, so that the data stored in the key-value database is consistent with the user-side data. When the user provides the key of the user data to query the user data, the KV entry corresponding to the updated user data can be used to obtain the updated user data, which helps to improve the accuracy of the data query.

[0149] In an embodiment of the present application, the memory is used to store a computer program and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0150] In the embodiment of the present application, the processor can be any hardware processing device that can execute the logic of the above method. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it can also be a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD) and other programmable devices; or an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (System on Chip, SOC), etc., but not limited to this.

[0151] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology or other technologies.

[0152] In an embodiment of the present application, a power supply component is configured to provide power to various components of the device in which it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0153] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0155] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0160] The storage medium of a computer is a readable storage medium, which may also be referred to as a readable medium. The readable storage medium includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0162] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, It is characterized in that include: Determine the data to be migrated; Migrate the data to be migrated from the original storage medium to the target storage medium; Determine, from the key-value database, a first key-value entry corresponding to the data to be migrated of the original storage medium; Generate a second version number based on the first version number included in the first key-value entry; Generate a second key-value entry corresponding to the to-be-migrated data of the target storage medium using the key included in the first key-value entry as the key, the first major version number included in the first key-value entry as the major version number, and the second major version number as the minor version number; as well as, In the process of migrating the data to be migrated from the original storage medium to the target storage medium, obtaining a data update request for the data to be migrated; Acquire target data for replacing the data to be migrated from the data update request; Updating the data to be migrated in the original storage medium to the target data; Based on the first major version number, generate a second major version number; A third key-value entry corresponding to the target data is generated using the key included in the first key-value entry as the key, the second major version number as the major version number, and the set minor version number as the minor version number.

2. The method according to claim 1, It is characterized in that The step of generating a second key-value entry corresponding to the data to be migrated of the target storage medium using the key included in the first key-value entry as the key, the first major version number included in the first key-value entry as the major version number, and the second major version number as the minor version number includes: The second key-value entry is generated using the key contained in the first key-value entry as the key, the first major version number contained in the first key-value entry as the major version number, the second major version number as the minor version number, and the address information of the target storage medium as the value.

3. The method according to claim 1, It is characterized in that The generating a second version number based on the first version number included in the first key-value entry includes: The first version number is increased by a set gradient to obtain the second version number.

4. The method according to any one of claims 1 to 3, It is characterized in that Also includes: Obtaining a first data query request; Obtaining a key to be queried from the first data query request; Using the key to be queried to perform matching in key-value entries in a key-value database, to obtain a first target key-value entry containing the key to be queried; Determine, from the first target key-value entries, a second target key-value entry with the largest major version number; Determine a third target key-value entry with the largest minor version number from the second target key-value entries; According to the third target key value entry, obtaining data corresponding to the key to be queried; A query result is determined based on the data corresponding to the key to be queried.

5. The method according to any one of claims 1 to 3, It is characterized in that Also includes: Get the specified reserved major version number corresponding to the specified key; From the key-value entries containing the specified key, obtain a fourth target key-value entry containing a major version number that is less than or equal to the specified reserved major version number; Determine a fifth target key-value entry with the largest major version number from the fourth target key-value entries; Determine a sixth target key-value entry with the largest minor version number from the fifth target key-value entry; The sixth target key-value entry is persisted as a snapshot of the key-value entry corresponding to the specified key.

6. The method according to claim 5, It is characterized in that Also includes: Obtaining a second data query request; In the case where the key to be queried contained in the second data query request is the specified key, matching is performed in the key-value entries using the specified key to obtain a seventh target key-value entry containing the specified key; the seventh target key-value entry contains the sixth target key-value entry; If the seventh target key-value entry contains an eighth target key-value entry whose major version number is greater than that of the sixth target key-value entry, when the data corresponding to the eighth target key-value entry is missing, the data corresponding to the specified key is obtained according to the sixth target key-value entry; Based on the data corresponding to the specified key, a query result is determined.

7. A computing device, It is characterized in that include: A memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps in the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing computer instructions, It is characterized in that When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 1 to 6.

9. A computer program product, It is characterized in that include: A computer program; when the computer program is executed by a processor, the processor is caused to execute the steps in the method according to any one of claims 1 to 6.

Images