A data processing method, apparatus, device, and storage medium
By utilizing the free space of subkeys in the KKV storage model to write data, the problem of low data write performance is solved, and more efficient data writing is achieved.
Patent Information
- Application Number
- CN202510629888.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
In table structures based on the KKV storage model, data write performance is poor due to the continuous incrementing subkey characteristics, which leads to frequent data movement.
In the key-value storage structure of the target data table, determine whether there is a free subkey interval at the write location. If it exists, directly determine the subkey from the free interval and write the data; if it does not exist, determine the target subkey free interval and the data to be moved through the cursor object, and reallocate the subkey to avoid data movement.
By avoiding data migration, data writing performance is improved, and data writing efficiency is increased.
Smart Images

Figure CN123115125A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology
[0002] Key-Key-Value (KKV) storage model is an extended key-value storage model widely used in data storage systems. In addition to a primary key, the KKV storage model includes a subkey (Subordinate Key, also known as a secondary key), which is used to organize data at a finer granularity based on the primary key.
[0003] Currently, table structures based on the KKV storage model typically use continuously increasing integers as subkeys. Due to this continuously increasing characteristic of the subkeys, data migration is inevitable when writing data to the middle position of the table structure (between adjacent entries), resulting in poor data writing performance. Summary of the Invention
[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a data processing method, apparatus, device and storage medium.
[0005] In a first aspect, embodiments of this disclosure provide a data processing method, the method comprising:
[0006] Receive a data write request for a target data table; wherein the data write request carries the data to be written and the write position information;
[0007] In the key-value storage structure of the target data table, it is determined whether there is a free subkey interval at the write position indicated by the write position information; wherein, the key-value storage structure belongs to the data structure type of primary key, subkey and value, the data type of the subkey is an integer of a preset length, and the key-value storage structure supports the existence of free intervals between adjacent subkeys;
[0008] If it is determined that there is a free subkey interval at the write position in the key-value storage structure, then a first subkey is determined for the data to be written from the free subkey interval, and the data to be written is written to the key-value storage structure as the value corresponding to the first subkey.
[0009] In an optional implementation, after determining whether a subkey free interval exists at the write position indicated by the write position information in the key-value storage structure of the target data table, the method further includes:
[0010] If it is determined that there is no free subkey interval at the write position in the key-value storage structure, then the target free subkey interval and the data to be moved are determined based on the adjacent subkeys corresponding to the write position; wherein, the target free subkey interval is determined based on the number of data moves and the length of the free interval;
[0011] Determine a second subkey for the data to be written from the free space of the target subkey, and determine a third subkey for the data to be moved;
[0012] The data to be written is written into the key-value storage structure as the value corresponding to the second subkey, and the data to be moved is moved to the value corresponding to the third subkey.
[0013] In one optional implementation, determining the target subkey free interval and the data to be moved based on the adjacent subkeys corresponding to the write position includes:
[0014] For the first direction and the second direction indicated by the adjacent subkeys corresponding to the write position, a first cursor object and a second cursor object are generated respectively; wherein, the first cursor object is used to determine the subkey free interval and the number of data moves by moving in the first direction, and the second cursor object is used to determine the subkey free interval and the number of data moves by moving in the second direction;
[0015] Based on the length of the subkey free space and the number of data moves determined by the first cursor object and the second cursor object respectively, the target subkey free space and the data to be moved are determined.
[0016] In one optional implementation, determining the target subkey free area and the data to be moved based on the length of the subkey free area and the number of data moves determined by the first cursor object and the second cursor object respectively includes:
[0017] By synchronously moving the first cursor object and the second cursor object in the first direction and the second direction respectively, the subkey free interval and the number of data moves are determined;
[0018] When the ratio between the length of the determined subkey free interval and the number of data moves is greater than a preset threshold, the subkey free interval is determined as the target subkey free interval, and the data to be moved is determined based on the number of data moves.
[0019] In one optional implementation, there is a free interval between each pair of the second subkey and the third subkey, the free interval including one or more subkey spaces, each of the one or more subkey spaces supporting the determination of a subkey.
[0020] In one optional implementation, there is a free interval between the first subkey and its adjacent subkeys, the free interval including one or more subkey spaces, each of the one or more subkey spaces supporting the determination of a subkey.
[0021] In one optional implementation, the subkey of the first data written in the key-value storage structure of the target data table is an intermediate value of an integer of the preset length.
[0022] Secondly, this disclosure provides a data processing apparatus, the apparatus comprising:
[0023] The receiving module is used to receive a data write request for a target data table; wherein the data write request carries the data to be written and the write position information;
[0024] The first determining module is used to determine whether there is a free subkey interval at the write position indicated by the write position information in the key-value storage structure of the target data table; wherein, the key-value storage structure belongs to the data structure type of primary key, subkey and value, the data type of the subkey is an integer of a preset length, and the key-value storage structure supports the existence of free intervals between adjacent subkeys;
[0025] The second determining module is used to determine a first subkey for the data to be written from the subkey free interval when it is determined that there is a subkey free interval at the write position in the key-value storage structure, and write the data to be written as the value corresponding to the first subkey into the key-value storage structure.
[0026] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the data processing method provided in embodiments of this disclosure.
[0027] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program for performing the data processing method provided in embodiments of this disclosure.
[0028] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.
[0029] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:
[0030] In the data processing method provided in this embodiment, after receiving a data write request for a target data table, it is determined in the key-value storage structure of the target data table whether there is a free subkey interval at the write position indicated by the data write request. If there is a free subkey interval at the write position, a first subkey is directly determined from the free subkey interval, and the data to be written is written to the key-value storage structure of the target data table as the value corresponding to the first subkey. Since the data type of the subkey in the key-value storage structure of the target data table in this embodiment is an integer of a preset length, and it supports free intervals between adjacent subkeys, data migration can be largely avoided when writing data between adjacent entries in the target data table, thereby improving the performance of data writing. Attached Figure Description
[0031] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0032] Figure 1 This is a schematic diagram of a table structure based on the KKV storage model provided in related technologies;
[0033] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this disclosure;
[0034] Figure 3 A schematic diagram of a target database provided in an embodiment of this disclosure;
[0035] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this disclosure;
[0036] Figure 5 A schematic diagram of yet another target database provided in an embodiment of this disclosure;
[0037] Figure 6 A schematic diagram of another target database provided in an embodiment of this disclosure;
[0038] Figure 7 A schematic diagram of yet another target database provided in an embodiment of this disclosure;
[0039] Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure;
[0040] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0041] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0042] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0043] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0044] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0045] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0046] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0047] The Key-Key-Value (KKV) storage model, consisting of a primary key, a subkey, and a value, is an extended key-value storage model widely used in data storage systems. For example... Figure 1 The diagram shown illustrates a table structure based on the KKV storage model provided in related technologies. This table structure includes a primary key MyList and multiple subkeys under this primary key. These subkeys are consecutively increasing integers -1, 0, 1, 2, and each subkey has a corresponding value a, b, c, d. Figure 1As can be seen, when writing data x to the middle position of the table structure (assuming it is between subkey -1 and subkey 1), data migration will occur. That is, the value b corresponding to subkey 0 and the value a corresponding to subkey -1 need to be migrated, which leads to poor data writing performance.
[0048] To address this, this disclosure provides a data processing method. Upon receiving a data write request for a target data table, the method determines whether a free subkey interval exists at the write position indicated by the write request in the key-value storage structure of the target data table. If a free subkey interval exists at the write position, a first subkey is directly determined from the free subkey interval, and the data to be written is written to the key-value storage structure of the target data table as the value corresponding to the first subkey. Since the data type of the subkeys in the key-value storage structure of the target data table in this disclosure is an integer of a preset length, and free intervals exist between adjacent subkeys, data migration can be largely avoided when writing data between adjacent entries in the target data table, thereby improving data write performance.
[0049] Specifically, this disclosure provides a data processing method, such as... Figure 2 The diagram shown is a flowchart illustrating a data processing method provided in an embodiment of this disclosure. This method can be executed by a data processing device, which can be implemented using software and / or hardware and is generally integrated into an electronic device. The data processing method provided in this embodiment can be applied to a data processing device, which can be deployed in a database or data storage system. Figure 2 As shown, the method includes:
[0050] S201: Receive a data write request for the target data table.
[0051] The data write request carries the data to be written and the write location information.
[0052] In this embodiment of the disclosure, the target data table can be any data table that has a data writing requirement. The target data table is a data table created based on the KKV storage model. The KKV storage model includes three parts: primary key, subkey, and value. The target data table can have a primary key. Each entry in the target data table corresponds to a subkey in the KKV storage model, and the data stored in each entry is the value corresponding to the subkey.
[0053] The data write request in this embodiment is a request to insert or delete data in a target database. The data write request includes the data to be written and its write position information. The data to be written can refer to the data that needs to be written to the target data table. The write position information indicates the write position of the data to be written in the target data table, and can indicate a specific position in the target data table. For example, the write position information can be between the Nth and Mth entries in the target data table.
[0054] In this embodiment of the disclosure, when a data write request for a target data table is received, the data to be written and the write position information of the data to be written are obtained by analyzing the data write request, and the data to be written is written to the target data table according to the write position information.
[0055] S202: In the key-value storage structure of the target data table, determine whether there is a subkey free interval at the write position indicated by the write position information.
[0056] The key-value storage structure is a data structure type consisting of a primary key, a subkey, and a value. The data type of the subkey is an integer of a preset length. The key-value storage structure supports the existence of free intervals between adjacent subkeys.
[0057] The key-value storage structure of the target data table in this embodiment is a data structure type built on the KKV storage model. This key-value storage structure includes three fields: primary key, subkey, and value. The primary key can be used to identify different data sets, such as different data tables. Within a data set identified by a primary key, the subkey is used to further refine the organization of the data in that data set.
[0058] In this embodiment, the data type of the subkey can be an integer of a preset length. That is, the data type of the subkey is a pre-set fixed-length integer type. Using an integer of a preset length as the subkey in the KVV storage model does not have the characteristic that consecutive subkeys must be continuously incremented in the related technology. Therefore, the key-value storage structure in this embodiment can support the existence of free intervals between adjacent subkeys in the key-value storage structure, so as to reduce the probability of data migration when writing data.
[0059] In the key-value storage structure of this disclosure, the data type of the subkey can be an unsigned 64-bit integer (uint64), an unsigned 32-bit integer (uint32), or other integer types with a fixed length.
[0060] like Figure 3The diagram illustrates a target data table provided in an embodiment of this disclosure. In this target data table, the user ID can be used as the primary key, with ID_1 serving as the primary key. Multiple subkeys can be included under the ID_1 primary key, each with a corresponding value. Under the ID_1 primary key, a subkey is created for each video work published by user ID_1, where 0x8000000 000000ff can correspond to a video work published by user ID_1.
[0061] In the key-value storage structure of this disclosure, the value field can be used to store specific data. For example, as shown in the above example, the values of each video work published by user ID_1, or the links of each video work, can be used as the values of their corresponding subkeys.
[0062] In the key-value storage structure of this disclosure, the free interval of the subkey refers to the free interval between the subkeys of adjacent entries in the target data table. This free interval can be used to determine a new subkey. In other words, the subkeys of adjacent entries are not contiguous, and the free interval of the subkey can be the interval used to determine a new subkey.
[0063] As mentioned above Figure 3 As shown, when a data write request is received for data d at both ends of the target data table (assuming it is after subkey 0x8000000000000ff), it can be known that there is only an adjacent subkey 0x8000000000000ff at that write position, i.e., there is a free subkey interval. When a data write request is received for data x between adjacent entries of the target data table (assuming it is subkey 0x7fffffff ffffffff and subkey 0x8000000 000000ff), it can be known that there is a free interval between subkey 0x7fffffff ffffffff and 0x8000000 000000ff, i.e., there is a difference between subkey 0x7fffffff ffffffff and 0x8000000000000ff.
[0064] In this embodiment of the disclosure, when a data write request for a target data table is received, the write position information in the data write request is first obtained. Then, in the key-value storage structure of the target data table, it is determined whether there is a free subkey interval based on the write position indicated by the write position information. If there is a free subkey interval, a new subkey can be directly determined for the data to be written in the free subkey interval, so that the data to be written is written in the value field corresponding to the new subkey without the need for data migration.
[0065] S203: If it is determined that there is a free subkey interval at the write position in the key-value storage structure, then a first subkey is determined for the data to be written from the free subkey interval, and the data to be written is written to the key-value storage structure as the value corresponding to the first subkey.
[0066] In this embodiment of the disclosure, when a free subkey interval is determined in the key-value storage structure of the target data table based on the write position, a subkey can be determined from the free subkey interval as the first subkey, and then the data to be written is written to the key-value storage structure as the value corresponding to the first subkey. The first subkey can be any subkey within the free subkey interval.
[0067] In one optional implementation, to minimize the probability of data migration during subsequent data writing, the first subkey determined for the data to be written within the subkey free interval also needs to have a free interval between it and its adjacent subkeys. This free interval can include one or more subkey spaces, and multiple subkeys can be determined within each subkey space. That is, the free interval between the first subkey determined for the data to be written and its adjacent subkeys includes at least one subkey space within which new subkeys can be determined for the data to be written. For example, this free interval is 256.
[0068] As mentioned above Figure 3 As shown, within the free interval of the subkey, there is a free interval between the first subkey 0x8000000000001ff, which is determined for the data to be written d, and the adjacent subkey 0x8000000 000000ff.
[0069] This disclosure provides a data processing method. Upon receiving a data write request for a target data table, the method determines whether a free subkey interval exists at the write position indicated by the write request in the key-value storage structure of the target data table. If a free subkey interval exists at the write position, a first subkey is directly determined from the free subkey interval, and the data to be written is written to the key-value storage structure of the target data table as the value corresponding to the first subkey. Since the data type of the subkeys in the key-value storage structure of the target data table in this disclosure is an integer of a preset length, and free intervals exist between adjacent subkeys, data migration can be largely avoided when writing data between adjacent entries in the target data table, thereby improving data write performance.
[0070] This disclosure also provides a data processing method, such as... Figure 4 The diagram shown is a flowchart illustrating another data processing method provided in this embodiment of the present disclosure. The method includes:
[0071] S401: Receives a data write request for the target data table.
[0072] The data write request carries the data to be written and the write location information.
[0073] S402: In the key-value storage structure of the target data table, determine whether there is a subkey free interval at the write position indicated by the write position information.
[0074] The key-value storage structure is a data structure type consisting of a primary key, subkeys, and values. The data type of the subkeys is an integer of a preset length. The key-value storage structure supports free intervals between adjacent subkeys. The descriptions of S401-S402 can be found in the above embodiments and will not be repeated here.
[0075] S403: If it is determined that there is no free subkey interval at the write position in the key-value storage structure, then the target free subkey interval and the data to be moved are determined based on the adjacent subkeys corresponding to the write position.
[0076] The target subkey free interval is determined based on the number of data moves and the length of the free interval.
[0077] In this embodiment of the disclosure, after receiving a data write request for a target data table, it is determined in the key-value storage structure of the target data table whether there is a subkey free interval at the write position indicated by the write position information carried in the data write request. If it is determined that there is no subkey free interval at the write position in the key-value storage structure, it is necessary to determine the target subkey free interval and the data to be moved based on the adjacent subkeys corresponding to the write position. The target subkey free area can be used to write the data to be moved and the data to be written.
[0078] like Figure 5 The diagram shows another target data table provided in this embodiment of the present disclosure. When a data write request is received for writing data y to the middle position of the target data (between subkey 0x8000000 00000000 and subkey 0x8000000 00000001), it can be determined through subkey 0x8000000 00000000 and subkey 0x8000000 00000001 that there is no free subkey interval at this write position.
[0079] Since there is no free subkey space at the write location in this key-value storage structure, there is data to be moved during this data write. The data to be moved refers to the data that needs to be moved from the original subkey to the subkey that has been redefined. It can be used to redefine the target subkey space range of the subkey. It is determined by the number of data moves and the space range length determined in two directions of the subkey in the key-value storage structure. The number of data moves is the number of data moves assumed to be moved in a certain subkey direction. The space range length can refer to the size of the available range contained in the free subkey space.
[0080] In one optional implementation, the target subkey free space and the data to be moved can be determined by creating cursor objects at adjacent subkeys corresponding to the write position. Specifically, a first cursor object and a second cursor object are generated for the first direction and the second direction indicated by the adjacent subkeys corresponding to the write position, respectively. The first direction and the second direction can refer to two different directions defined by the adjacent subkeys corresponding to the write position. Specifically, the first direction can be assumed to be in front of or to the left of an adjacent subkey, and the second direction can be assumed to be behind or to the right of another adjacent subkey; these two directions are different. The first cursor object is used to determine the subkey free space and the number of data moves by moving in the first direction, and the second cursor object is used to determine the subkey free space and the number of data moves by moving in the second direction.
[0081] In other words, during the movement of the first and second cursor objects along their respective subkey directions, the subkey space interval and the number of data transfers are calculated.
[0082] In one optional implementation, a preset movement amount is first set. Then, based on the preset movement amount, the first and second cursor objects are moved synchronously along a first direction and a second direction. During the movement of the two cursor objects along the first and second directions, the free interval of the subkeys is calculated based on their respective adjacent subkeys and the original subkeys corresponding to the current movement position, and the current number of data moves is counted. The preset movement amount is a predefined number of original subkeys that the first and second cursor objects need to move each time, and the number of data moves can be initialized to 0.
[0083] As mentioned above Figure 5As shown, assuming the preset movement amount is 1, and the adjacent subkeys corresponding to the write position are known to be 0x80000000000000 and 0x8000000 0000001, a cursor object p1 is created at the adjacent subkey 0x8000000 00000000, and a cursor object p2 is created at the adjacent subkey 0x8000000 00000001. Then, according to the preset movement amount, cursor objects p1 and p2 move forward and backward synchronously along the subkey direction, respectively. Taking cursor object p2 as an example, when cursor object p2 moves backward to the original subkey 0x8000000 00000006, the data shift count increases by 1. The length of the free interval between the adjacent subkey 0x8000000 00000001 and the original subkey 0x8000000 00000006 corresponding to the current position is calculated, and the length of the subkey free interval is found to be 5.
[0084] Based on the above embodiments, the target subkey free area and the data to be moved are determined according to the length of the subkey free area and the number of data moves determined by the first cursor object and the second cursor object, respectively. Specifically, it can be determined whether the ratio between the length of the subkey free area and the number of data moves is greater than a preset threshold to determine whether the above-mentioned subkey free area is taken as the target subkey free area, and the data to be moved is determined according to the above-mentioned number of data moves. The preset threshold is a pre-set value, which is used to measure whether the above-mentioned subkey free area can ensure that there is at least a preset threshold number of subkey spaces between adjacent subkeys when the above-mentioned number of data moves are moved. That is, when the number of times the data needs to be moved is the above-mentioned number of data moves, this preset threshold can ensure that there is at least a preset threshold number of subkey spaces between adjacent subkeys in the above-mentioned subkey free area. In other words, the subkey spaces between adjacent subkeys in the above-mentioned subkey free area are specifically determined by the preset threshold.
[0085] In one application scenario, during the synchronous movement of the first cursor object and the second cursor object along the first direction and the second direction respectively, the ratio between the length of the subkey free interval determined by the two cursor objects in their respective subkey directions and the current number of data moves is calculated. If the ratio between the length of the subkey free interval determined by one of the cursor objects in its corresponding subkey direction and the current number of data moves is greater than a preset threshold, then the subkey free interval is determined as the target subkey free interval, and the data to be moved is determined based on the number of data moves.
[0086] As mentioned above Figure 5As shown, assuming the preset threshold is 1, when the cursor object p2 moves backward to the original subkey position 0x800000000000006, the data migration count is incremented by 1, and the length of the subkey free interval is calculated to be 5. Then, the ratio between the length of the subkey free interval and the number of data migrations is calculated to see if it exceeds the preset threshold. If the ratio exceeds the preset threshold, the subkey free interval can be determined as the target subkey free interval, and the data to be migrated can be determined based on the number of data migrations.
[0087] In another optional implementation, based on the above embodiments, when the first cursor object and the second cursor object move synchronously along the first direction and the second direction respectively, since the number of data moves for the first cursor object and the second cursor object is determined by a preset movement amount (i.e., the number of data moves for the first cursor object and the second cursor object is the same), when the ratio between the length of the subkey free interval determined by the two cursor objects in their respective subkey directions and the current number of data moves is greater than a preset threshold, the lengths of the subkey free intervals determined by the two cursor objects in their respective subkey directions can be compared to determine the target subkey free interval. Specifically, the subkey free interval with the larger length can be used as the target subkey free interval.
[0088] As can be seen, when the ratio between the length of the subkey free interval determined by the first cursor object and the second cursor object in their respective directions and the number of data moves is greater than the preset threshold, the target subkey free interval can be determined by comparing the length of the subkey free interval determined by the two cursor objects in their respective subkey directions. This can largely ensure that there are more free intervals between adjacent subkeys, thereby avoiding data moves to a large extent when writing data between adjacent entries in the target data table, thus reducing moves and further improving the performance of data writing.
[0089] In another optional implementation, during the synchronous movement of the first and second cursor objects along the first and second directions respectively, if one of the cursor objects moves to the first or last entry in the target data table, the subkey free interval in that subkey direction can be determined as the target subkey free interval, and the data to be moved can be determined based on the number of data moves. Here, the cursor object moving to the first or last entry in the target data table can indicate the existence of an infinite number of subkey free intervals.
[0090] like Figure 6The diagram illustrates another target data table provided in this embodiment. When a data write request is received for data y to be written to the middle position of the target data table (between subkeys 0x8000000 00000000 and 0x8000000 00000001), it is known that there is no free subkey interval at this write position. Therefore, based on the adjacent subkeys 0x8000000 00000000 and 0x8000000 00000001 corresponding to the write position, a cursor object p1 is created at subkey 0x8000000 00000000, and a cursor object p2 is created at subkey 0x8000000 00000001. Then, according to a preset movement amount, cursor objects p1 and p2 move along subkey 0x8000000 00000000000000000000001. The cursor object p2 moves synchronously in the forward direction of 00000000 and the backward direction of subkey 0x800000000000001. When the cursor object p2 moves to the last entry position in the target data table, the subkey free interval determined by the cursor object p2 can be directly used as the target subkey free interval, and the data to be moved is determined based on the number of data moves.
[0091] As can be seen, during the movement of cursor objects based on the direction indicated by adjacent subkeys, if one of the cursor objects moves to the position of the first or last entry in the target data table, the subkey free interval determined by that cursor object can be directly used to determine the target subkey free interval.
[0092] In another optional implementation, during the synchronous movement of the first and second cursor objects along the first and second directions respectively, if one of the cursor objects moves to the value boundary of the subkey, the target subkey free interval and the number of data moves can be determined by the subkey free interval determined by the cursor object in the other subkey direction. Specifically, the cursor object moving to the subkey boundary (i.e., the boundary position of the value range of the subkey's data type, which is also the boundary of an integer of a preset length) indicates that there is currently no additional subkey free interval. In this case, the target subkey free interval and the data to be moved need to be determined by the cursor object in the other subkey direction.
[0093] like Figure 7The diagram illustrates another target data table provided in this embodiment. When a data write request is received for writing data y to the middle position of the target data table (between subkey 0xfffffffffffffffe and subkey 0xffffffffffff), it is known that there is no free subkey interval at this write position. Based on the adjacent subkeys 0xfffffffffffffffe and 0xffffffffffffff, cursor object p1 is created at subkey 0xfffffffffffffffe, and cursor object p2 is created at subkey 0xffffffffffffffff. Then, according to a preset movement amount, cursor objects p1 and p2 move forward along the direction of subkey 0xfffffffffffffffffe and subkey 0xffffffffffffffffff, respectively. The cursor object p2 moves synchronously in the backward direction. When the cursor object p2 moves to the value boundary (0xffffffffffffffffff) of the data type of the subkey, the target subkey free interval and the data to be moved can be determined based on the subkey free interval determined by another cursor object p1.
[0094] As can be seen, during the movement of the cursor object based on the direction indicated by the adjacent subkey, if the cursor object in one subkey direction moves to the extreme position of the value range of the data type of the subkey, the free interval of the target subkey and the data to be moved can be determined by the cursor object in the other subkey direction.
[0095] S404: Determine a second subkey for the data to be written from the free space of the target subkey, and determine a third subkey for the data to be moved.
[0096] After determining the target subkey free range and the data to be moved, a second subkey is determined from the target subkey free range for the data to be written, and a third subkey is determined from the target subkey free range for the data to be moved. The second and third subkeys can be any subkey within the target subkey free range.
[0097] In one optional implementation, to minimize data migration when writing data between adjacent entries in the target data table, a free interval can be created between the second and third subkeys when determining the second subkey for the data to be written and the third subkey for the data to be migrated from the free interval of the target subkey. This free interval can include one or more subkey spaces, and each subkey space can support the determination of multiple subkeys. In other words, the free interval between the determined second and third subkeys can accommodate multiple subkeys. By reserving this free interval, new data can be fully utilized to store the free interval when writing data to adjacent entries in the target data table, thereby effectively reducing the frequency of data migration operations due to insufficient space and improving data writing performance.
[0098] As mentioned above Figure 5 As shown, after determining the target subkey free interval and the data to be moved based on the cursor object, a second subkey 0x8000000 00000002 is determined for the data to be written y from the target subkey free interval, and a third subkey 0x8000000 00000004 is determined for the data to be moved c. There are multiple free intervals between the second subkey 0x8000000 00000002 and the third subkey 0x8000000 00000004, and multiple subkeys can be determined within each free interval.
[0099] S405: Write the data to be written as the value corresponding to the second subkey into the key-value storage structure, and move the data to be moved to the value corresponding to the third subkey.
[0100] As mentioned above Figure 5 As shown, the data to be written, y, is written to the value corresponding to the second subkey 0x8000000 00000002, and the data to be moved, c, is moved to the value corresponding to the third subkey 0x8000000 00000004.
[0101] In the data processing method provided in this embodiment, after receiving a data write request for a target data table, it is determined whether there is a free subkey interval at the write position indicated by the data write request in the key-value storage structure of the target data table. If there is no free subkey interval at the write position, a target free subkey interval and the data to be moved are determined based on the adjacent subkeys corresponding to the write position. Then, a second subkey and a third subkey are determined from the target free subkey interval, and the data to be written and the data to be moved are written into the key-value storage structure of the target data table.
[0102] As can be seen, when it is determined that there is no free subkey interval at the write position, the embodiments of this disclosure support the method of determining the target free subkey interval and the data to be moved based on the adjacent subkeys corresponding to the write position, and then determining the second and third subkeys. This can also greatly reduce the probability of subsequent data movement and further improve the performance of data writing.
[0103] In one optional implementation, when a data write request for a target data table is received, if it is detected that the data to be written carried in the data write request is the first data to be written in the key-value storage structure of the target data table, then a subkey can be arbitrarily determined from the data value range of the subkey for the data to be written, as the starting subkey of the key-value storage structure.
[0104] In another optional implementation, when a data write request for the target data table is received, if it is detected that the data to be written carried in the data write request is the first data to be written in the key-value storage structure of the target data table, then the middle value of an integer of a preset length can be used as the subkey of the data to be written. By selecting the middle value of an integer of a preset length as the subkey of the first data to be written in the key-value storage structure of the target data table, subsequent data writes to the target data table can be distributed as evenly as possible on both sides of the starting subkey, thereby minimizing the probability of subsequent data migration.
[0105] To implement the above embodiments, this disclosure also proposes a data processing apparatus. Figure 8 This is a schematic diagram of a data processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 8 As shown, the device includes:
[0106] The receiving module 801 is used to receive a data write request for a target data table; wherein the data write request carries the data to be written and the write position information.
[0107] The first determining module 802 is used to determine whether there is a free subkey interval at the write position indicated by the write position information in the key-value storage structure of the target data table; wherein, the key-value storage structure belongs to the data structure type of primary key, subkey and value, the data type of the subkey is an integer of a preset length, and the key-value storage structure supports the existence of free intervals between adjacent subkeys;
[0108] The second determining module 803 is used to determine a first subkey for the data to be written from the subkey free interval when it is determined that there is a subkey free interval at the write position in the key-value storage structure, and write the data to be written as the value corresponding to the first subkey into the key-value storage structure.
[0109] In one optional embodiment, the apparatus further includes:
[0110] The third determining module is used to determine the target subkey free interval and the data to be moved based on the adjacent subkeys corresponding to the write position when it is determined that there is no subkey free interval at the write position in the key-value storage structure; wherein, the target subkey free interval is determined according to the number of data moves and the length of the free interval;
[0111] The fourth determining module is used to determine a second subkey for the data to be written from the free interval of the target subkey, and to determine a third subkey for the data to be moved;
[0112] The data writing module is used to write the data to be written as the value corresponding to the second subkey into the key-value storage structure, and to move the data to be moved to the value corresponding to the third subkey.
[0113] In one optional implementation, the third determining module includes:
[0114] A generation submodule is used to generate a first cursor object and a second cursor object for the first direction and the second direction indicated by the adjacent subkeys corresponding to the write position, respectively; wherein, the first cursor object is used to determine the subkey free interval and the number of data moves by moving in the first direction, and the second cursor object is used to determine the subkey free interval and the number of data moves by moving in the second direction;
[0115] The first determination submodule is used to determine the target subkey free area and the data to be moved based on the length of the subkey free area and the number of data moves determined by the first cursor object and the second cursor object, respectively.
[0116] In one optional implementation, the first determining submodule includes:
[0117] The second determination submodule is used to determine the subkey free interval and the number of data moves by synchronously moving the first cursor object and the second cursor object in the first direction and the second direction, respectively.
[0118] The third determination submodule is used to determine the subkey free interval as the target subkey free interval when the ratio between the length of the determined subkey free interval and the number of data moves is greater than a preset threshold, and to determine the data to be moved based on the number of data moves.
[0119] In one optional implementation, there is a free interval between each pair of the second subkey and the third subkey, the free interval including one or more subkey spaces, each of the one or more subkey spaces supporting the determination of a subkey.
[0120] In one optional implementation, there is a free interval between the first subkey and its adjacent subkeys, the free interval including one or more subkey spaces, each of the one or more subkey spaces supporting the determination of a subkey.
[0121] In one optional implementation, the subkey of the first data written in the key-value storage structure of the target data table is an intermediate value of an integer of the preset length.
[0122] In the data processing apparatus provided in this embodiment, after receiving a data write request for a target data table, it determines whether there is a free subkey interval at the write position indicated by the data write request in the key-value storage structure of the target data table. If there is a free subkey interval at the write position, a first subkey is directly determined from the free subkey interval, and the data to be written is written to the key-value storage structure of the target data table as the value corresponding to the first subkey. Since the data type of the subkey in the key-value storage structure of the target data table in this embodiment is an integer of a preset length, and it supports free intervals between adjacent subkeys, data migration can be largely avoided when writing data between adjacent entries in the target data table, thereby improving the performance of data writing.
[0123] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the data processing method described in this disclosure.
[0124] This disclosure also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, implement the data processing method described in this disclosure.
[0125] In addition, this disclosure also provides an electronic device, see [link to relevant documentation]. Figure 9 As shown, it may include:
[0126] The electronic device includes a processor 901, a memory 902, an input device 903, and an output device 904. The number of processors 901 in the electronic device can be one or more. Figure 9 Taking a processor as an example. In some embodiments of this disclosure, the processor 901, memory 902, input device 903, and output device 904 can be connected via a bus or other means, wherein, Figure 9 Taking the example of a connection between China and Israel via a bus.
[0127] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The input device 903 can be used to receive input digital or character information, and to generate signal inputs related to user settings and function control of the electronic device.
[0128] Specifically in this embodiment, the processor 901 loads the executable files corresponding to the processes of one or more applications into the memory 902 according to the following instructions, and the processor 901 runs the applications stored in the memory 902 to realize the various functions of the above-mentioned electronic device.
[0129] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0130] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, characterized in that, include: Receive a data write request for a target data table; wherein the data write request carries the data to be written and the write position information; In the key-value storage structure of the target data table, it is determined whether there is a free subkey interval at the write position indicated by the write position information; wherein, the key-value storage structure belongs to the data structure type of primary key, subkey and value, the data type of the subkey is an integer of a preset length, and the key-value storage structure supports the existence of free intervals between adjacent subkeys; If it is determined that there is a free subkey interval at the write position in the key-value storage structure, then a first subkey is determined for the data to be written from the free subkey interval, and the data to be written is written to the key-value storage structure as the value corresponding to the first subkey.
2. The data processing method according to claim 1, characterized in that, After determining whether a subkey free interval exists at the write position indicated by the write position information in the key-value storage structure of the target data table, the method further includes: If it is determined that there is no free subkey interval at the write position in the key-value storage structure, then the target free subkey interval and the data to be moved are determined based on the adjacent subkeys corresponding to the write position; wherein, the target free subkey interval is determined based on the number of data moves and the length of the free interval; Determine a second subkey for the data to be written from the free space of the target subkey, and determine a third subkey for the data to be moved; The data to be written is written into the key-value storage structure as the value corresponding to the second subkey, and the data to be moved is moved to the value corresponding to the third subkey.
3. The data processing method according to claim 2, characterized in that, The step of determining the target subkey free interval and the data to be moved based on the adjacent subkeys corresponding to the write position includes: For the first direction and the second direction indicated by the adjacent subkeys corresponding to the write position, a first cursor object and a second cursor object are generated respectively; wherein, the first cursor object is used to determine the subkey free interval and the number of data moves by moving in the first direction, and the second cursor object is used to determine the subkey free interval and the number of data moves by moving in the second direction; Based on the length of the subkey free space and the number of data moves determined by the first cursor object and the second cursor object respectively, the target subkey free space and the data to be moved are determined.
4. The data processing method according to claim 3, characterized in that, The step of determining the target subkey free area and the data to be moved based on the length of the subkey free area and the number of data moves determined by the first cursor object and the second cursor object respectively includes: By synchronously moving the first cursor object and the second cursor object in the first direction and the second direction respectively, the subkey free interval and the number of data moves are determined; When the ratio between the length of the determined subkey free interval and the number of data moves is greater than a preset threshold, the subkey free interval is determined as the target subkey free interval, and the data to be moved is determined based on the number of data moves.
5. The data processing method according to claim 2, characterized in that, There are free intervals between the second subkey and the third subkey, and the free intervals include one or more subkey spaces, each of the one or more subkey spaces supporting the determination of a subkey.
6. The data processing method according to claim 1, characterized in that, There is a free interval between the first subkey and its adjacent subkeys. The free interval includes one or more subkey spaces, and each subkey space in the one or more subkey spaces supports the determination of a subkey.
7. The data processing method according to claim 1, characterized in that, The subkey of the first data written in the key-value storage structure of the target data table is the middle value of the integer of the preset length.
8. A data processing apparatus, characterized in that, The device includes: The receiving module is used to receive a data write request for a target data table; wherein the data write request carries the data to be written and the write position information; The first determining module is used to determine whether there is a free subkey interval at the write position indicated by the write position information in the key-value storage structure of the target data table; wherein, the key-value storage structure belongs to the data structure type of primary key, subkey and value, the data type of the subkey is an integer of a preset length, and the key-value storage structure supports the existence of free intervals between adjacent subkeys; The second determining module is used to determine a first subkey for the data to be written from the subkey free interval when it is determined that there is a subkey free interval at the write position in the key-value storage structure, and write the data to be written as the value corresponding to the first subkey into the key-value storage structure.
9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-7.
11. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-7.
