Data processing method, data query method, device, equipment, medium and product

Abstract

The embodiments of the present application provide a data processing method, a data query method, an apparatus, a device, a medium, and a product, which can be applied to the field of computer technology. The data processing method includes: in response to a write instruction or a disk flush instruction, writing the data to be stored indicated by the write instruction to an active memory table stored in the internal memory, or writing the data to be stored indicated by the disk flush instruction to a first file stored in the first-level external memory, the active memory table stores multiple data in the order of multiple keys, the data includes a key with a volume identifier, and the first file corresponding to the active memory table includes at least one data with each of the multiple volume identifiers; in response to a merge operation between the n-1th level external memory and the nth level external memory, obtaining target volume data with a target volume identifier among the multiple volume identifiers, and writing the target volume data with the target volume identifier to the nth file corresponding to the target volume identifier stored in the nth level external memory, where n is an integer greater than 1.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a data processing method, a data query method, an apparatus, a device, a medium, and a product. Background Art

[0002] With the development of computer storage technology, data storage systems based on Log Structured MergeTree (LSM-Tree) are becoming increasingly popular. They can be applied to key-value storage databases, distributed file systems, or time series databases, and are particularly suitable for large-scale data storage systems that need to optimize both write throughput and query latency.

[0003] In order to meet the respective requirements of write operations and query operations, it is necessary to reduce write latency and improve query efficiency. Summary of the Invention

[0004] In view of the above problems, embodiments of the present application provide a data processing method, a data query method, an apparatus, a device, a medium, and a product that reduce write latency and improve query efficiency.

[0005] According to a first aspect of an embodiment of the present application, a data processing method is provided, which may include: in response to a write instruction or a disk flush instruction, writing the to-be-stored data indicated by the above-mentioned write instruction to an active memory table stored in the internal memory, or writing the to-be-stored data indicated by the above-mentioned disk flush instruction to a first file stored in a first-level external memory, wherein the above-mentioned active memory table stores multiple data in the order of multiple keys, the above-mentioned data includes a key with a volume identifier, and the first file corresponding to the above-mentioned active memory table includes at least one data with each of the multiple volume identifiers; in response to a merge operation between the n-1th level external memory and the nth level external memory, obtaining target volume data with a target volume identifier among the multiple volume identifiers, and writing the target volume data with the above-mentioned target volume identifier to the nth file corresponding to the above-mentioned target volume identifier stored in the above-mentioned n-level external memory, wherein n is an integer greater than 1.

[0006] According to the second aspect of an embodiment of the present application, a data query method is provided, which may include: obtaining a query instruction, wherein the query instruction includes a key to be queried having a volume identifier to be queried; in response to not finding a query value matching the key to be queried from active memory tables and inactive memory tables stored in the memory, and at least one first file stored in a first-level external memory according to the key to be queried, querying a query value matching the key to be queried from an nth external memory file corresponding to the volume identifier to be queried and stored in an nth-level external memory, where n is an integer greater than 1.

[0007] According to the third aspect of an embodiment of the present application, a data processing device is provided, which may include: a first writing module, configured to write the to-be-stored data indicated by the above-mentioned writing instruction to the active memory table stored in the internal memory in response to a writing instruction or a disk flushing instruction, or write the to-be-stored data indicated by the above-mentioned disk flushing instruction to the first file stored in the first-level external memory, wherein the above-mentioned active memory table stores multiple data in the order of multiple keys, the above-mentioned data includes a key with a volume identifier, and the first file corresponding to the above-mentioned active memory table includes at least one data with each of the multiple volume identifiers; an obtaining module, configured to obtain target volume data with a target volume identifier among the multiple volume identifiers in response to a merge operation between the n-1th level external memory and the nth level external memory; and a second writing module, configured to write the target volume data with the above-mentioned target volume identifier to the nth file corresponding to the above-mentioned target volume identifier stored in the above-mentioned n-level external memory, wherein n is an integer greater than 1.

[0008] According to the fourth aspect of an embodiment of the present application, a data query device is provided, which may include: an acquisition module configured to acquire a query instruction, wherein the query instruction includes a key to be queried having an identifier of a volume to be queried; a query module configured to query for a query value matching the key to be queried from an nth external storage file corresponding to the volume identifier to be queried and stored in an nth-level external storage, in response to not finding a query value matching the key to be queried from active memory tables and inactive memory tables stored in the memory, and at least one first file stored in a first-level external storage according to the key to be queried, where n is an integer greater than 1.

[0009] According to a fifth aspect of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0010] According to the sixth aspect of the present application, a computer-readable storage medium is also provided, on which a computer program or instruction is stored. When the above computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0011] According to the seventh aspect of the present application, a computer program product is also provided, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0013] Figure 1A schematic diagram of a storage device for storing log-structured merged data is shown;

[0014] Figure 2 A schematic diagram illustrating a data processing method based on a hybrid-separated storage architecture according to an embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram showing the principles of a splitting policy for target volume data and a retention policy for other data according to an embodiment of the present application is shown;

[0016] Figure 4 A system architecture diagram of a data processing system applicable to a data processing method or a data query method according to an embodiment of the present application is shown;

[0017] Figure 5 A flow chart of a data processing method according to an embodiment of the present application is shown;

[0018] Figure 6 A schematic diagram illustrating the application of a splitting policy for target volume data and a retention policy for other data according to an embodiment of the present application is shown;

[0019] Figure 7 A flow chart of a data query method according to an embodiment of the present application is shown;

[0020] Figure 8 A block diagram of a data processing device according to an embodiment of the present application is shown;

[0021] Figure 9 A block diagram of a data query device according to an embodiment of the present application is shown;

[0022] Figure 10 A block diagram of an electronic device suitable for implementing a data processing method or a data query method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, A, B, and C, etc.).

[0027] In the embodiment of the present application, "indication" may include direct indication, indirect indication, explicit indication or implicit indication. In the case of describing that a certain indication information is used to indicate A, it can be understood that the indication information carries A, directly indicates A or indirectly indicates A.

[0028] In the embodiments of the present application, the various numerical numbers involved are only used for the convenience of description and are not used to limit the scope of protection of this application. The size of the serial numbers involved in the embodiments of the present application does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic. For example, the terms "first", "second", "third", "fourth" and other various terminology labels (if any) in the description, claims and drawings of the embodiments of the present application can be used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. Among them, the terms used in this way are interchangeable where appropriate.

[0029] Unless otherwise specified or there is no logical conflict, the terms and / or descriptions between different embodiments of the present application are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0030] In order to clearly describe the technical solutions of the embodiments of the present application, some terms involved in the embodiments of the present application are explained below.

[0031] 1. Write amplification, read amplification, space amplification, write-intensive applications, read-intensive applications

[0032] Write amplification may refer to the ratio between the amount of data actually written to the storage device and the amount of data actually written to the storage device when writing data.

[0033] Read amplification may refer to the ratio between the amount of data actually read from a storage device and the amount of data actually read from the storage device when querying data.

[0034] Space amplification may refer to the ratio between the actual occupied storage space and the logical size of valid data.

[0035] Write-intensive applications are applications where write operations dominate their input / output operations. For example, write-intensive applications include logging applications. Write-intensive applications require reducing write amplification and improving write throughput.

[0036] Read-intensive applications refer to applications where read operations dominate input and output operations.

[0037] 2. Off-site update and local update

[0038] An out-of-place update refers to an update that doesn't affect the data itself, but instead writes the updated data to a new storage location. This data is retained and marked as invalid. Out-of-place updates are suitable for write-intensive scenarios.

[0039] In-place updates can refer to updates that directly affect the data itself, that is, the data to be updated can be updated based on the data to be updated. The data to be updated is stored in the same location as the data to be updated.

[0040] 3. Volume (or storage volume), volume ID, key with volume ID, volume data

[0041] A volume may refer to a logical partition unit for data storage. A volume identifier may indicate a volume. A key with a volume identifier may mean that the key includes the volume identifier. The volume identifier is a component of the key. Volume data may include at least one data item with a volume identifier. Different volume data items may have different volume identifiers. For example, a key may be represented as "<volume identifier, key identifier>".

[0042] 4. Log structure merge tree, memory, external memory, active memory table, inactive memory table, n-th level external memory, n-th file

[0043] Log-structured merge trees convert random writes into sequential writes through memory structures, which has high write performance. They also use logs to achieve data persistence, making them more user-friendly for write-intensive applications and capable of efficiently processing large-scale data. Storage devices used to store log-structured merge trees can include memory and external storage. Figure 1 Provide explanation.

[0044] Figure 1A schematic diagram of a storage device for storing log-structured merged data is shown.

[0045] like Figure 1 As shown, the storage device may include a memory and an external memory. The memory may be used to store an active memory table (i.e., MemTable) and at least one inactive memory table (i.e., Immutable MemTable).

[0046] The external memory may include multiple levels of external memory. For example, the external memory may include a first level of external memory (i.e., L0), a second level of external memory (i.e., L1), ..., an nth level of external memory (i.e., Ln-1), .... n may be an integer greater than or equal to 1. The nth level of external memory may be used to store at least one nth file. The number of files stored in different levels of external memory may be different. The number of nth files stored in the nth level of external memory may be greater than the number of n-1th files stored in the n-1th level of external memory. The file format of the nth file may be a sorted string table (SSTable).

[0047] 5. Hybrid storage, grouping by volume, hybrid layers, and separation layers

[0048] Hybrid storage can refer to data being stored in a global key-ordered manner, regardless of volume identifier. In embodiments of the present application, the active memory table, the inactive memory table, and the first file can be stored in a hybrid manner. The keys in the active memory table, the inactive memory table, and the first file can be referred to as global keys.

[0049] Volume storage may refer to distinguishing volume identifiers and storing data in order by local key. In an embodiment of the present application, the storage method for the nth file may be volume storage. The key in the nth file may be referred to as a local key. n may be an integer greater than 1.

[0050] The hybrid layer may refer to a layer of data stored in a hybrid storage manner. In an embodiment of the present application, the hybrid layer may include the internal memory and the first-level external memory.

[0051] The separation layer may refer to a layer of data stored in a volume storage manner. In the embodiment of the present application, the separation layer may include the second-tier external storage and the tier external storage above the second-tier external storage.

[0052] 6. Skip List

[0053] A skip list is a probabilistic data structure. It maintains an ordered structure similar to a multi-layer linked list. When data is written to the table, new nodes are added to the skip list. The height of a skip list node is at least 1, and its value can be determined based on probability. Nodes at higher levels have a lower probability of being generated, resulting in fewer nodes at higher levels and more nodes between adjacent higher-level nodes. When performing a query, the skip list starts at the top and works its way down. The upper-level linked list results in the skip list can be used as indexes for key queries, allowing higher-level nodes to skip over unmatched nodes, reducing the number of nodes to be queried and speeding up the query process.

[0054] With the development of computer storage technology, data storage systems based on log-structured merge trees are becoming more and more popular. They can be applied to key-value storage databases, distributed file systems, or time series databases, and are particularly suitable for large-scale data storage systems that need to optimize both write throughput and query latency.

[0055] A data storage system based on a log-structured merge tree may include the following storage architecture.

[0056] As an implementation approach, a fully hybrid storage architecture is used. For example, from the active in-memory table to the nth file, no distinction is made between volumes or columns. Because queries require traversing files stored in each tier of external storage, read amplification is a serious issue, resulting in low query efficiency and high input and output overhead.

[0057] Another implementation approach is to completely separate the storage architecture. For example, from the active in-memory table to the nth file, each is separated into volumes or columns. Because write operations require first determining the data classification of the data to be stored, that is, determining which volume or column to write the data to, this increases write latency and processor overhead.

[0058] In the process of realizing the inventive concept of this application, it was found that it was difficult to simultaneously meet the requirements of low write latency and improve query efficiency. For this reason, the embodiment of this application proposes a storage architecture based on hybrid-separation, that is, a hybrid storage method can be used in the hybrid layer and a volume storage method can be used in the separation layer. Hybrid storage can refer to not distinguishing volume identifiers, and data is stored by global keys. The hybrid layer can include memory and first-level external storage. Volume storage can refer to distinguishing volume identifiers, and data is stored by local keys. The separation layer can include second-level external storage and external storage at levels above the second-level external storage. In this way, query efficiency can be improved on the basis of improving write performance. For ease of understanding, the following is combined with Figure 2 Provide explanation.

[0059] Figure 2 A schematic diagram of the principles of a data processing method based on a hybrid-separate storage architecture according to an embodiment of the present application is shown.

[0060] like Figure 2As shown, in a hybrid-separate storage architecture, the hybrid layer may include memory and first-tier external storage. The hybrid layer may store data using a hybrid storage method. The memory may be used to store active memory tables and inactive memory tables. The first-tier external storage may be used to store at least one first file.

[0061] The active memory table may include multiple data. The data may include a key with a volume identifier, i.e., the volume identifier is a component of the key. Thus, the active memory table may include at least one data with each of multiple volume identifiers. The active memory table may store multiple data in an ordered order of multiple keys. The keys in the active memory table may be referred to as global keys. For example, Figure 2 The active memory table may include 8 data. Each of the 8 data may have "volume identification V1", "volume identification V2", "volume identification V1", "volume identification V1", "volume identification V1", "volume identification V3", "volume identification V1" and "volume identification V1".

[0062] An inactive memory table can be obtained by performing a state switch operation on an active memory table. The inactive memory table stores data and uses the same storage methods as the active memory table. Specifically, an inactive memory table can store multiple data items in a sequence of keys with volume identifiers. Similarly, the keys in an inactive memory table are called global keys.

[0063] The data stored in the first file may be obtained by performing a disk flushing operation on the inactive memory table. The first file may store data and storage methods in the same manner as the inactive memory table, that is, the first file may store multiple data in the order of multiple keys with volume identifiers.

[0064] The separation layer in the hybrid-separate storage architecture may include a second-tier external memory, ..., an nth-tier external memory, .... n may be an integer greater than 1. The separation layer may store data using a volume storage method. The nth-tier external memory may be used to store at least one nth file. The at least one nth file may have a respective nth file identifier. The at least one nth file identifier may each have a target volume identifier from a plurality of volume identifiers, i.e., the nth file identifier of the nth file corresponding to the target volume identifier may have the target volume identifier. Thus, the nth file corresponding to the target volume identifier may store target volume data having the target volume identifier. The key in the nth file may be referred to as a local key.

[0065] For example, Figure 2The target volume identifier in the file can be "Volume Identifier V1," "Volume Identifier V2," or "Volume Identifier V3." Thus, the nth file corresponding to the target volume identifier "Volume Identifier V1" can store the target volume data with "Volume Identifier V1." The nth file corresponding to the target volume identifier "Volume Identifier V2" can store the target volume data with "Volume Identifier V2." The nth file corresponding to the target volume identifier "Volume Identifier V3" can store the target volume data with "Volume Identifier V3."

[0066] Because the hybrid layer uses a hybrid storage method (i.e., a completely flattened storage method) for data storage, data belonging to different volumes are arranged consecutively in physical storage and share the same index structure. Therefore, write operations do not require pre-data classification and can simply append the data to be stored to the currently active memory table. In other words, the data to be stored can be written to the currently active memory table using an off-site update method. This simplifies the write path, thereby reducing write latency and improving write throughput. Because the separation layer uses a volume-based storage method for data storage, query operations can query the nth file corresponding to the volume identifier to be queried when accessing the second-tier external storage and external storage above the second-tier external storage, without having to query all nth files. This allows for accurate positioning of the nth file corresponding to the volume identifier to be queried, thereby reducing input and output overhead and improving query efficiency.

[0067] In addition, it was also found that due to the fully hybrid storage architecture, hot data with high frequency access is dispersedly stored in different n-th files, thus reducing the cache hit rate. Since the completely separated architecture may make the active memory tables of different columns or volumes meet the conditions for conversion to inactive memory tables at the same time to perform disk flushing operations, it leads to write amplification. Since the second-level external memory and the external memory above the second-level external memory in the embodiment of the present application adopt a storage method of volume storage, the hot data with high frequency access can be stored in a centralized manner, thereby improving the cache hit rate. Since the active memory table and the inactive memory table in the embodiment of the present application are mixedly stored, write amplification is alleviated.

[0068] In addition, it is also found that there is a storage fragmentation problem caused by premature splitting of highly mixed small data blocks. The small data block may refer to volume data with a stored data amount less than or equal to a first predetermined data amount.

[0069] To this end, an embodiment of the present application proposes that for the merge operation between the first-tier external storage and the second-tier external storage, a splitting strategy for the target volume data can be adopted, that is, the target volume data determined from the multiple volume data stored in the first-tier external storage is written to the second-tier external storage. The target volume data may refer to volume data that meets a predetermined condition. The predetermined condition may refer to a condition that is adapted to reduce storage fragmentation. For example, the predetermined condition may include at least one of a first predetermined data volume or a predetermined data volume ratio. The data volume of the target volume data may be greater than or equal to the first predetermined data volume or the data volume ratio of the target volume data may be greater than or equal to at least one of the predetermined data volume ratios. Thereby, the probability of storage fragmentation problems caused by premature splitting of highly mixed small data blocks is reduced. In addition, space amplification can be reduced, saving storage space.

[0070] In addition, the embodiment of the present application also proposes a retention policy for other data that does not meet the predetermined conditions, that is, other data in multiple volume data can be retained in the first-tier external storage. Since the first-tier external storage stores other data, the amount of data stored in the first-tier external storage is reduced.

[0071] In order to facilitate the understanding of the splitting strategy for target volume data and the retention strategy for other data, the following Figure 3 Provide explanation.

[0072] Figure 3 A schematic diagram showing the principles of a splitting policy for target volume data and a retention policy for other data according to an embodiment of the present application is shown.

[0073] like Figure 3 As shown, at least one data item having multiple volume identifiers can be obtained from at least one first file stored in the first-level external storage to obtain multiple volume data items. Target volume data items that meet predetermined conditions can then be determined from the multiple volume data items and written to a second file having the target volume identifier and stored in the second-level external storage item. The remaining data items in the multiple volume data items can be retained in the first-level external storage item. For example, a merge operation can be performed on the remaining data items in the multiple volume data items to obtain first merged data items. The first merged data items can then be merged into the target first file stored in the first-level external storage item.

[0074] The above describes the inventive concept of the embodiment of the present application. The data processing method and data query method provided by the embodiment of the present application are described in detail below with reference to the accompanying drawings.

[0075] First, combine Figure 4 The data processing system to which the data processing method is applicable is described. Figure 4 As an example, the system architecture shown in Figure 5 and Figure 6, the data processing method provided in the embodiment of the present application is specifically described. Figure 5 The data processing method is explained. Figure 6 Describe the splitting policy for target volume data and the retention policy for other data. Figure 7 The data query method provided in the embodiment of the present application is described in detail.

[0076] It should be noted that the embodiments of the present application can be implemented independently or in combination with each other, and the same or similar concepts or processes will not be described in detail in some embodiments.

[0077] Figure 4 A system architecture diagram of a data processing system applicable to a data processing method or a data query method according to an embodiment of the present application is shown.

[0078] like Figure 4 As shown, the data processing system may include a computing device and a storage device. In some embodiments, the computing device may be a server, a server cluster consisting of multiple servers, or a distributed system. The computing device may also be a cloud service cluster that provides basic cloud computing services such as cloud storage and cloud services, cloud databases, cloud computing, cloud functions, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, big data or artificial intelligence platforms, etc., which are not limited in the embodiments of the present application.

[0079] Computing devices may include memory and a processor. Memory can be used to store active and inactive memory tables. Memory can include at least one of the following: DRAM (Dynamic Random Access Memory), NVM (Non-Volatile Memory), off-heap memory, or persistent memory (PMEM).

[0080] The storage device may include external memory. The external memory may include at least one of the following: a solid-state drive (SSD), a hard disk drive (HDD), NVMe (Non-Volatile Memory Express), or persistent memory. Optionally, the external memory may also be a disk array. The external memory may include multiple levels of external memory, namely, first-level external memory, second-level external memory, ..., nth-level external memory, .... n may be an integer greater than 1. The nth-level external memory may be used to store the nth file. n may be an integer greater than 1.

[0081] For the description of the mixing layer and the separation layer, please refer to the description of the corresponding parts above and will not be repeated here.

[0082] The computing device can be used to execute the data processing method or data query method described in the embodiments of the present application. The data processing method may include a write operation for writing data to be stored in an active memory table, a state switching operation for an active memory table, a disk flushing operation for an inactive memory table, and a merge operation between the n-1th level external memory and the nth level external memory. Please refer to the detailed description of each operation below and will not be repeated here.

[0083] It should be noted that the data processing method or data query method described in the embodiments of the present application can also be executed by devices other than computing devices, and the embodiments of the present application are not limited to this. The execution entities of the data processing method and the data query method can be the same or different. Based on the technical solutions provided in the embodiments of the present application, more or fewer components or functions can be configured for the data processing system according to business needs.

[0084] The following will be based on Figure 4 The system architecture described, combined with Figure 5 、 Figure 6 and Figure 7 The data processing method and data query method of the embodiments of the present application are described in detail.

[0085] Figure 5 A flow chart of a data processing method according to an embodiment of the present application is shown.

[0086] like Figure 5 As shown, the method includes operations S510 to S520. Operation S510 involves a write operation and a disk flush operation. Operation S520 involves a merge operation.

[0087] In operation S510, in response to a write instruction, the data to be stored indicated by the write instruction is written to the active memory table stored in the memory, or in response to a flush instruction, the data to be stored indicated by the flush instruction is written to the first file stored in the first-level external memory.

[0088] In operation S520, in response to a merge operation between the n-1th level external storage and the nth level external storage, target volume data having a target volume identifier among multiple volume identifiers is obtained, and the target volume data having the target volume identifier is written to the nth file corresponding to the target volume identifier stored in the nth level external storage.

[0089] The data to be stored may be data that needs to be written to the active memory table or the first file. In the case where the data to be stored is data that needs to be written to the active memory table, the write instruction may include the data to be stored. Optionally, the write instruction may include a first access path, whereby the first access path may be accessed to obtain the data to be stored. In the case where the data to be stored is data that needs to be written to the first file, the flush instruction may include data in the inactive memory table. Optionally, the flush instruction may include a second access path, whereby the inactive memory table may be accessed according to the second access path. Thus, the data in the inactive memory table may be obtained. The data in the inactive memory table may be the data to be stored.

[0090] The target volume data may refer to volume data that meets a predetermined condition. The predetermined condition may include at least one predetermined evaluation metric. Optionally, the predetermined condition may refer to a condition used by a pre-trained classification model to determine whether the volume data is the target volume data.

[0091] The active memory table and the first file can be stored in a hybrid storage mode. The nth file can be stored in a volume-based storage mode. n can be an integer greater than 1. Hybrid storage can refer to data being stored in a global key-ordered manner, regardless of volume identifier. Volume-based storage can refer to data being stored in a local key-ordered manner, regardless of volume identifier. Hybrid storage and volume-based storage are described below.

[0092] The active memory table may include multiple data items. The data items may include a key with a volume identifier, i.e., the volume identifier is a component of the key. Thus, the active memory table may include at least one data item with each of the multiple volume identifiers. The multiple data items may be sorted according to the order of the multiple keys to obtain the multiple data items stored in the active memory table. The multiple data items are stored in the active memory table according to the order of the multiple keys. The keys in the active memory table may be referred to as global keys.

[0093] The inactive memory table can be obtained by performing a state switching operation on the active memory table. For example, if the data volume of the active memory table is equal to the third predetermined data volume, the state identifier of the active memory table can be configured as a read-only identifier to obtain an inactive memory table. Thus, the inactive memory table can be an active memory table whose state identifier is a read-only identifier and whose data volume is equal to the third predetermined data volume. The inactive memory table stores data and stores data in the same manner as the active memory table, that is, the inactive memory table can store multiple data in the order of multiple keys with volume identifiers. Similarly, the keys in the inactive memory table can be called global keys.

[0094] The data stored in the first file may be obtained by performing a disk flushing operation on an inactive memory table. The number of first files may be at least one. If the number of first files is one, the first file stores data and uses the same storage method as the inactive memory table, i.e., the first file may store multiple data in the order of multiple keys with volume identifiers. If the number of first files is multiple, the first file may store partial data from the inactive memory table and use the same storage method as the inactive memory table. Similarly, the key in the first file may be referred to as a global key.

[0095] The n-th level external memory can store at least one n-th file. The n-th file can be obtained by performing a merge operation between the n-1-th level and the n-th level. At least one n-th file can have its own n-th file identifier. At least one n-th file identifier can each have a target volume identifier from a plurality of volume identifiers, that is, the n-th file identifier of the n-th file corresponding to the target volume identifier can have a target volume identifier. Thus, the n-th file corresponding to the target volume identifier can store target volume data having the target volume identifier. n can be an integer greater than 1. The target volume data can include multiple data having the target volume identifier. The data can include a key having the target volume identifier. The key in the n-th file can be called a local key.

[0096] Based on the above, the storage path for the data stored in the first file can be at least one first file in the active memory table or the inactive memory table. Since the active memory table can include at least one data item with each of M volume identifiers, if one first file is included, the first file corresponding to the active memory table can include at least one data item with each of the M volume identifiers. If multiple first files are included, the first file corresponding to the active memory table can store partial data from the active memory table, and the partial data can be data with each of the M volume identifiers. Alternatively, the partial data can be data with each of some of the M volume identifiers. M can be an integer greater than 1.

[0097] According to an embodiment of the present application, since the active memory table can store multiple data items in the order of multiple keys, including keys with volume identifiers, a hybrid storage method is implemented for the active memory table. Based on this, when writing the data to be stored indicated by a write instruction to the active memory table, no prior data classification is required, thereby reducing write latency and increasing write throughput, thereby improving write performance. Furthermore, since the first file corresponding to the active memory table includes at least one data item with multiple volume identifiers, a hybrid storage method is implemented for the first file in the first-tier external storage. Since the nth file corresponding to the target volume identifier in the nth-tier external storage is used to store target volume data with the target volume identifier, per-volume storage of the nth file is implemented. Furthermore, since the nth file utilizes per-volume storage, a query value matching the query key can be retrieved from the nth external storage file corresponding to the query volume identifier stored in the nth-tier external storage, without having to query nth files with other volume identifiers. This improves query accuracy and efficiency, and reduces input and output overhead. n can be an integer greater than 1.

[0098] The following describes in detail the write operations, state switching operations, disk flushing operations, and merge operations between the first-level external memory and the second-level external memory involved in the hybrid layer, as well as the merge operations between the n-1th level external memory and the nth level external memory involved in the separation layer, where n∈{3, 4, ...}.

[0099] 1. Mixed layer

[0100] 1.1. Write operations involving the hybrid layer

[0101] In order to simplify the write path, the data to be stored can be directly appended to the active memory table without first determining the volume data to be stored from multiple volume data based on the volume identifier to be stored, and then determining the local to-be-written location from the volume data to be stored based on the local to-be-stored key, and then writing the data to be stored to the local to-be-written location. In this way, the data to be stored can be written to the volume data to be stored in the active memory table, thereby reducing the storage overhead generated by data classification (i.e., determining the volume data to be stored) and improving storage efficiency and write throughput.

[0102] Appending the data to be stored directly to the active memory table can be achieved in the following ways.

[0103] As an implementation method, the write instruction may include data to be stored. The data to be stored may include a value to be stored and a key to be stored having a volume identifier to be stored. For example, the key to be stored may be expressed as "<volume identifier to be stored, key identifier to be stored>". The computing device may receive a write instruction. According to the key to be stored having a volume identifier to be stored included in the write instruction, a global location to be written is determined from the active memory table. The key to be stored may be referred to as a global key to be stored. Thus, the data to be stored may be written to the global location to be written. For example, an atomic operation may be used to write the data to be stored to the global location to be written, thereby writing the data to be stored to the active memory table. In addition, the computing device may also write the data to be stored to a write-ahead log (WAL) stored in an external memory to maintain data persistence.

[0104] As an implementation, the computing device may include a write thread, whereby the write thread may be used to perform a write operation, that is, the write thread may be used to receive a write instruction, and determine a global write location from an active memory table according to a key to be stored included in the write instruction.

[0105] The data structure of the active memory table can be configured based on actual business needs and is not limited here. For example, the data structure of the active memory table can be a skip list, a balanced binary search tree, or a B+ tree. A balanced binary search tree can include at least one of the following: a red-black tree or an AVL tree (Adelson-Velsky and Landis Tree). The following uses the skip list as an example to further explain how to write data to the active memory table.

[0106] Skip lists can include multiple layers of linked lists. The linked list on the current layer can be a subset of the linked list on the next layer. The lowest layer of the linked list can include multiple data stored in the active memory table. Higher-level linked lists can use randomization to determine whether data is located in a higher-level linked list. Skip lists allow queries to skip some data, thereby accelerating queries. A node can contain data and a pointer to the same key in the next layer of the linked list. Data can include values ​​and keys with volume identifiers.

[0107] As an implementation method, traversal starts from the highest level linked list of the jump list. For the current level linked list among the multiple level linked lists, traverse multiple nodes of the current level linked list, and for the current node among the multiple nodes, when the key of the current node is equal to the key to be stored, update the value of the current node according to the value to be stored. When the key of the current node is greater than the key to be stored or the current node reaches the end of the current level linked list, move to the next level linked list, and the next level linked list can be used as the new current level linked list. Repeat the above traversal operation until the lowest level linked list is reached. Determine the predecessor node of the position to be written in the lowest level linked list. Generate a new node, which may include the data to be stored. Randomly determine the layer to which the new node belongs, and insert the new node into each layer.

[0108] 1.2. State switching operation

[0109] The computing device can obtain the amount of data stored in the active memory table. When the amount of data stored in the active memory table equals a third predetermined amount of data, the state flag of the active memory table is configured as a read-only flag, thereby converting the active memory table into an inactive memory table. As an implementation, the computing device can include a write thread, and thus, the write thread can be used to perform a state switching operation. Specifically, in response to the amount of data stored in the active memory table equaling the third predetermined amount of data, the write thread is used to configure the state flag of the active memory table to a read-only flag, thereby converting the inactive memory table into an inactive memory table. Furthermore, the write thread can also be used to store the inactive memory table in a dynamic array.

[0110] 1.3. Flashing operation

[0111] A disk flushing operation serializes the data structure of an inactive memory table to external storage. The storage medium changes from the inactive memory table in memory to the nth file in external storage. The inactive memory table and the first file store the same data and use the same storage method: the inactive memory table stores multiple data items sorted by multiple keys. The first file can also store multiple data items sorted by multiple keys. The data structure of the same data in the inactive memory table and the first file can be different, while the content is the same. A disk flushing operation can achieve persistence of in-memory data, preserve mixed data, and change the storage medium.

[0112] As an implementation method, the disk-swiping instruction may be generated in the following manner: the disk-swiping instruction may be generated in response to the number of inactive memory tables being equal to a predetermined number. Optionally, the disk-swiping instruction may be generated in response to the amount of data stored in the first-level external memory being equal to a fourth predetermined data amount. Optionally, the disk-swiping instruction may be generated in response to the amount of data stored in the active memory table being greater than a third predetermined data amount. Optionally, the disk-swiping instruction may be generated in response to a triggering operation of an operating body. The operating body may include a user or a stylus. The predetermined number, the third predetermined data amount, and the fourth predetermined data amount may be configured according to actual business needs and are not limited here.

[0113] Thus, the computing device can access the inactive memory table in response to the disk flushing instruction according to the second access path indicated by the disk flushing instruction. Obtain the data to be stored from the inactive memory table and write the data to be stored to the first file according to a predetermined data format. For example, the computing device can scan the inactive memory table in sequence to obtain the data of the inactive memory table. The data of the inactive memory table is the data to be stored. The obtained data is serialized into a binary format to obtain the data stored in the first file. As an implementation method, the data structure of the inactive memory table can be a jump table. Thus, the computing device can scan the inactive memory table in the jump table order to obtain the data of the inactive memory table.

[0114] As an implementation, the computing device may include a disk flushing thread. Thus, the disk flushing thread performs a disk flushing operation, i.e., the disk flushing operation is performed in response to a disk flushing instruction, and the inactive memory table is accessed according to the second access path indicated by the disk flushing instruction. The data to be stored is obtained from the inactive memory table and written to the first file in a predetermined data format. Furthermore, the disk flushing thread, in response to completing the disk flushing operation on the inactive memory table, removes the inactive memory table from the dynamic array, thereby implementing management of the inactive memory table using dynamic data.

[0115] 1.4. Merge operation between first-tier and second-tier external storage

[0116] In response to a merge operation between the first-tier external memory and the second-tier external memory, the computing device can obtain at least one data item having multiple volume identifiers from at least one first file, thereby obtaining multiple volume data items. The at least one first file can be stored in the first-tier external memory. Target volume data meeting predetermined conditions is then determined from the multiple volume data items, and the target volume data is written to a second file having the target volume identifier stored in the second-tier external memory item, while the remaining data in the multiple volume data items can be retained in the first-tier external memory item. The target volume data items can be referred to as dominant volume data items. The following describes the target volume data splitting strategy and the retention strategy for the remaining data items in conjunction with the accompanying drawings.

[0117] 1.4.1. Splitting strategy for target volume data (or leading volume data)

[0118] Regarding how to obtain multiple volume data from at least one first file, it can be achieved in the following manner.

[0119] As an implementation method, the data may include a key with a volume identifier. The computing device may obtain at least one data with each of the multiple volume identifiers from at least one first file to obtain volume data with each of the multiple volume identifiers. For any volume identifier among the multiple volume identifiers included in the at least one first file, at least one data with the volume identifier may be obtained from the at least one first file to thereby obtain volume data with the volume identifier. Optionally, at least one data with the volume identifier may be obtained from the at least one first file, and deduplication may be performed on the at least one data with the volume identifier to obtain the volume identifier with the volume data. The volume data of each of the multiple volume identifiers may be obtained in the above manner.

[0120] Regarding how to determine target volume data from multiple volume data, it can be achieved in the following manner.

[0121] 1.4.1.1. Based on satisfying at least one predetermined evaluation condition

[0122] The predetermined condition may include at least one predetermined evaluation indicator. The at least one evaluation indicator may include at least one of the following: a first predetermined data volume, a predetermined data volume ratio, or a predetermined access frequency. The first predetermined data volume, the predetermined data volume ratio, and the predetermined access frequency may be configured according to actual business needs and are not limited here. For example, in order to match the system load, at least one of the first predetermined data volume, the predetermined data volume ratio, or the predetermined access frequency may be adjusted according to at least one of the load information or the inter-volume data distribution entropy. The load information may include at least one of the following: the data volume stored in the first-level external memory, the first resource utilization rate, or the first input-output delay. The inter-volume data distribution entropy may indicate the degree of uniformity of data distribution in multiple volumes.

[0123] As an implementation method, in response to at least one of the data amount stored in the first-level external memory being greater than or equal to the second predetermined data amount, the data distribution entropy between volumes being greater than or equal to the predetermined distribution entropy, the first resource utilization being greater than or equal to the predetermined utilization, or the first input-output delay being greater than or equal to the predetermined delay, at least one of the first predetermined data amount, the proportion of the predetermined data amount, or the predetermined access frequency is reduced.

[0124] As an implementation method, in response to at least one of the amount of data stored in the first-level external memory being less than the second predetermined data amount, the data distribution entropy between volumes being less than the predetermined distribution entropy, the first resource utilization being less than the predetermined utilization, or the first input and output delay being less than the predetermined delay, at least one of the first predetermined data amount, the proportion of the predetermined data amount, or the predetermined access frequency is increased.

[0125] The computing device may evaluate the plurality of volume data according to at least one predetermined evaluation metric to obtain evaluation results for each of the plurality of volume data. The evaluation results for the volume data may indicate whether the volume data meets the at least one predetermined evaluation metric.

[0126] Since the predetermined evaluation index for evaluating whether volume data is target volume data can be adjusted according to at least one of load information or inter-volume data distribution entropy, adaptive adjustment of the predetermined evaluation instruction is achieved, thereby improving the accuracy of determining the target volume data.

[0127] In response to each of the plurality of volume data including an evaluation result indicating that the volume data satisfies at least one predetermined evaluation metric, the volume data may be selected as the target volume data. For any of the plurality of volume data, in response to the evaluation result indicating that the volume data satisfies at least one predetermined evaluation metric, the volume data may be selected as the target volume data. In response to the evaluation result indicating that the volume data does not satisfy at least one predetermined evaluation metric, that is, in response to the evaluation result indicating that the volume data does not satisfy any predetermined evaluation metric, the volume data may not be selected as the target volume data. Thus, the plurality of volume data may not include target volume data. Alternatively, the plurality of volume data may include one target volume data or multiple target volume data.

[0128] When the at least one predetermined evaluation indicator includes at least one of the following: a first predetermined data volume, a predetermined data volume percentage, or a predetermined access frequency, the evaluation result of the target volume data may indicate at least one of the following: the data volume of the target volume data is greater than or equal to the first predetermined data volume, the data volume percentage of the target volume data is greater than or equal to the predetermined data volume percentage, or the access frequency of the target volume data within a predetermined time period is greater than or equal to the predetermined access frequency. The data volume percentage of the target volume data may be the ratio of the data volume of the target volume data to the total data volume of the multiple volumes of data.

[0129] Thus, the evaluation result of the target volume data may indicate that the data volume of the target volume data is greater than or equal to the first predetermined data volume. Optionally, the evaluation result of the target volume data may indicate that the data volume ratio of the target volume data is greater than or equal to the predetermined data volume ratio. Optionally, the evaluation result of the target volume data may indicate that the access frequency of the target volume data within a predetermined time period is greater than or equal to the predetermined access frequency. Optionally, the evaluation result of the target volume data may indicate that the data volume of the target volume data is greater than or equal to the first predetermined data volume and the data volume ratio of the target volume data is greater than or equal to the predetermined data volume ratio. Optionally, the evaluation result of the target volume data may indicate that the data volume of the target volume data is greater than or equal to the first predetermined data volume and the access frequency of the target volume data within the predetermined time period is greater than or equal to the predetermined access frequency. Optionally, the evaluation result of the target volume data may indicate that the data volume ratio of the target volume data is greater than or equal to the predetermined data volume ratio and the access frequency of the target volume data within the predetermined time period is greater than or equal to the predetermined access frequency. Optionally, the evaluation result of the target volume data may indicate that: the data volume of the target volume data is greater than or equal to the first predetermined data volume, the data volume proportion of the target volume data is greater than or equal to the predetermined data volume proportion, and the access frequency of the target volume data in the predetermined time period is greater than or equal to the predetermined access frequency.

[0130] The first predetermined data volume is used to reduce the probability of small data blocks being split, and the predetermined data volume ratio is used to reduce the probability of storage fragmentation, thereby reducing the probability of storage fragmentation caused by premature splitting of highly mixed small data blocks.

[0131] In the case of including multiple target volume data, the target volume data can be written to the second file stored in the second-tier external memory in the following manner.

[0132] As an implementation, the computing device may write respective target volume data to multiple second files stored in the second-tier external memory. The multiple second files each have a different target volume identifier. The target volume data may have a target volume identifier.

[0133] As another implementation, the computing device may determine a target volume data item from multiple target volume data items. For example, the target volume data item may be randomly determined from the multiple target volume data items. Alternatively, the target volume data item may be determined from the multiple target volume data items based on the priority of at least one predetermined evaluation metric and the evaluation results of each of the multiple target volume data items. The target volume data item has a target volume identifier. The target volume data item may be written to a second file having the target volume identifier and stored in the second-tier external memory.

[0134] 1.4.1.2. Based on pre-trained classification model

[0135] The target volume data can be determined by a computing device using a pre-trained classification model based on multiple volume data. The model structure of the pre-trained classification model can be configured based on actual business needs and is not limited here. Because the pre-trained classification model can achieve dynamic adaptation and self-learning, it can upgrade static methods to intelligent and adaptive solutions, thereby improving the accuracy of target volume data determination.

[0136] The pre-trained classification model can be obtained in the following way.

[0137] Sample volume characteristic data for each of the plurality of sample volume data can be obtained based on the plurality of sample volume data and the sample system data. The sample volume characteristic data can include at least one of the following: the amount of data included in the sample volume data, the data volume percentage of the sample volume data, a third resource utilization rate, or a third input / output delay. The data volume percentage of the sample volume data can be the ratio of the data volume of the sample volume data to the total data volume of the plurality of sample volume data. The sample volume characteristic data for each of the plurality of sample volume data is used to train an initial classification model to obtain a pre-trained classification model.

[0138] For example, for any sample volume data among a plurality of sample volume data, the sample volume feature data of the sample volume data can be input into the initial classification model to obtain the sample classification result of the sample volume data. In this way, the sample classification results of each of the plurality of sample volume data can be obtained. Based on a predetermined loss function, a loss function value is obtained according to the sample classification results of each of the plurality of sample volume data and the sample classification labels of each of the plurality of sample volume data. The model parameters of the initial classification model are adjusted according to the loss function value until a predetermined end condition is met, thereby obtaining a pre-trained classification model. The predetermined end condition may include at least one of the predetermined training rounds reaching the maximum training rounds or the loss function value being less than or equal to a predetermined threshold. The predetermined training rounds and the predetermined threshold can be configured according to actual business needs and are not limited here.

[0139] Since the sample volume feature data includes at least one of the following items: the amount of data included in the sample volume data, the data volume proportion of the sample volume data, the third resource utilization rate or the third output input delay, the comprehensiveness of the sample volume feature data is improved, and all of the feature data are related to whether the evaluation volume data is the target volume data. Therefore, the sample volume feature data is used to train the initial classification model, thereby improving the accuracy of the classification results of the pre-trained classification model.

[0140] Thus, the computing device can obtain volume feature data for each of the multiple volume data items based on the multiple volume data items and the system data. For any of the multiple volume data items, the volume feature data for that volume data item is input into a pre-trained classification model to obtain a classification result for that volume data item. Thus, a classification result for each of the multiple volume data items can be obtained. The classification result can indicate whether the volume data item is the target volume data item, that is, whether the volume data item is the target volume data item or not.

[0141] The computing device may designate the volume data as target volume data in response to the classification results of each of the plurality of volume data including a classification result indicating that the volume data is target volume data. For example, for any volume data among the plurality of volume data, the computing device may designate the volume data as target volume data in response to the classification result indicating that the volume data is target volume data. In response to the classification result indicating that the volume data is not target volume data, the computing device may not designate the volume data as target volume data. Thus, the plurality of volume data may not include target volume data. Alternatively, the plurality of volume data may include one or more target volume data.

[0142] In the case of including multiple target volume data, the target volume data can be written to the second file stored in the second-tier external memory in the following manner.

[0143] As an implementation, the computing device may write respective target volume data to multiple second files stored in the second-tier external memory. The multiple second files each have a different target volume identifier. The target volume data may have a target volume identifier.

[0144] As another implementation, the computing device may determine a target volume data from a plurality of target volume data. For example, the computing device may randomly determine a target volume data from the plurality of target volume data. The computing device may write the target volume data to a second file having the target volume identifier stored in the second-tier external memory.

[0145] 1.4.2 Retention Policy for Other Data

[0146] The computing device may obtain first merged data in response to a merge operation on data other than the target volume data in the at least one first file, and then write the first merged data to the target first file stored in the first-level external memory. The target first file may be the first file in the at least one first file. Optionally, the target first file may be a newly created file other than the at least one first file. The first merged data may include different volume identifiers for the at least two data items.

[0147] Since other data in at least one first file other than the target volume data is volume data that does not meet at least one predetermined evaluation indicator or is volume data determined not to be target volume data using a pre-trained classification model, a merge operation is performed on the other data to obtain first merged data, and the first merged data is written to the target first file stored in the first-level external storage, thereby achieving the retention of the target first file in the first-level external storage and reducing the amount of data stored in the first-level external storage through the merge operation.

[0148] In addition, the computing device may also be configured with a predetermined identifier associated with the target first file to reduce the probability of being processed repeatedly. The predetermined identifier may serve as metadata. The predetermined identifier may indicate that the target first file has participated in at least one of the rounds or merge sources of the merge operation, and does not participate in the predetermined number of merge operations after the round. Not participating in the predetermined number of merge operations after the round may refer to not participating in multiple rounds of merge operations after the round. The predetermined number of times may be configured according to actual business needs and is not limited here. For example, the predetermined number of times may be 5. The round in which the target first file has participated in the merge operation may be the second round, the merge source may be the first file SSTableA and the first file SSTableB, and the predetermined number of times may be 3. Thus, the target first file may not participate in the third, fourth, and fifth rounds of merge operations.

[0149] In order to better understand the splitting strategy for target volume data and the retention strategy for other data in the embodiment of the present application, Figure 6 Provide explanation.

[0150] Figure 6 A schematic diagram of the application of a splitting policy for target volume data and a retention policy for other data according to an embodiment of the present application is shown.

[0151] like Figure 6 As shown, the first file may include eight data items. The data items may include a key with a volume identifier. The volume identifiers of the eight data items are "Volume Identifier V1," "Volume Identifier V2," "Volume Identifier V1," "Volume Identifier V1," "Volume Identifier V1," "Volume Identifier V3," "Volume Identifier V2," and "Volume Identifier V1." The predetermined evaluation indicator may include a predetermined data volume ratio of 0.6.

[0152] For "Volume ID V1," "Volume ID V2," and "Volume ID V3," we obtain the volume data with "Volume ID V1," the volume data with "Volume ID V2," and the volume data with "Volume ID V3." The volume data with "Volume ID V1" accounts for 0.625% of the total data volume. The volume data with "Volume ID V2" accounts for 0.25% of the total data volume. The volume data with "Volume ID V3" accounts for 0.125% of the total data volume.

[0153] Because the volume data with volume identifier V1 accounts for a larger percentage than the predetermined percentage, the volume data with volume identifier V1 is the target volume data. Volume identifier V1 is the target volume identifier. The target volume data can be written to the second file corresponding to the target volume identifier, which is stored in the second-tier external storage.

[0154] In response to a merge operation on other data in the first file except the target volume data, merged data is obtained and written into the target first file stored in the first-tier external memory.

[0155] In addition, in order to further improve the query efficiency, when a similar first file is stored in the first-level external memory, if the predetermined identifier associated with the target first file indicates not to participate in the merge operation, the merge operation between the target first file and the similar first file can be performed to obtain the second merged data. The second merged data can be written to other first files stored in the first-level external memory. The similarity between the similar first file and the target first file can be greater than or equal to the predetermined similarity. The predetermined similarity can be configured according to actual business needs and is not limited here. The other first file can be the target first file. Optionally, the other first file can be a newly created file other than the target first file. The second merged data may include at least two data having different volume identifiers.

[0156] 2. Separation layer

[0157] At the separation layer, each volume data can be stored independently, and the volume data can be stored using a B+ tree. The nth file corresponding to the target volume identifier stores multiple data in the order of multiple keys with the target volume identifier. The data stored in the nth file corresponding to the target volume identifier all have the same target volume identifier. The key ranges between different nth files do not overlap. In addition, the computing device can also configure at least one of the storage medium or compression strategy adapted for the nth file. For example, the compression strategy can include at least one of the following: a high compression ratio strategy or a low compression ratio strategy. The file identifier of the nth file can have a target volume identifier. Since the compression strategy and storage medium can be configured according to actual business needs, different needs are met and scalability is improved.

[0158] 2.1. Merge operation between the n-1th level external memory and the nth level external memory involved in the separation layer

[0159] When n is an integer greater than 2, the computing device can obtain the n-1th target volume data having a target volume identifier among multiple volume identifiers from at least one n-1th file and obtain the nth target volume data having a target volume identifier from at least one nth file in response to a merge operation between the n-1th level external memory and the nth level external memory.

[0160] The n-1th file can be stored in the n-1th level external storage. The nth file can be stored in the nth level external storage. The n-1th target volume data and the nth target volume data may have overlapping key ranges. For example, the key range of the n-1th target volume data may be the n-1th key range. The key range of the nth target volume data may be the nth key range. The n-1th key range and the nth key range have overlapping key ranges.

[0161] The computing device may obtain target volume data having a target volume identifier based on the (n-1)th target volume data and the (n)th target volume data having the target volume identifier. For example, the computing device may perform a merge operation on the (n-1)th target volume data and the (n)th target volume data having the target volume identifier to obtain the target volume data having the target volume identifier.

[0162] Thus, the computing device can write the target volume data having the target volume identifier to the nth file corresponding to the target volume identifier stored in the nth level external memory.

[0163] Figure 7 A flow chart of a data query method according to an embodiment of the present application is shown.

[0164] like Figure 7 The method includes operations S710 to S720.

[0165] In operation S710 , a query instruction is acquired.

[0166] In operation S720, in response to the fact that a query value matching the key to be queried is not found from the active memory tables and the inactive memory tables stored in the memory, and from at least one first file stored in the first-level external memory, a query value matching the key to be queried is queried from the nth external memory file corresponding to the volume identifier to be queried and stored in the nth-level external memory.

[0167] n can be an integer greater than 1. In the data query method of an embodiment of the present application, the data stored in the inactive memory table can be different from the data stored in the active memory table, that is, there are different portions of data stored in the inactive memory table and the data stored in the active memory table. The active memory table can refer to a memory table that is currently in a modifiable state. The inactive memory table can refer to a memory table that is currently in a read-only state. The inactive memory table does not correspond to the active memory table, that is, the inactive memory table is not derived from the active memory table.

[0168] The query instruction may include a query key having an identifier of a volume to be queried. The computing device may determine whether a query value matching the query key is found in the order of the memory, the first-level external memory, and the last-level external memory, until a query value matching the query key is found or no query value matching the query key is found in the last-level external memory.

[0169] For example, the computing device may determine, based on the key to be queried, whether a query value matching the key to be queried is found from an active memory table stored in the memory. In response to finding a query value matching the key to be queried from the active memory table, a query result including the query value is obtained. In response to not finding a query value matching the key to be queried from the active memory table, the computing device may determine, based on the value to be queried, whether a query value matching the key to be queried is found from an inactive memory table stored in the memory. In response to finding a query value matching the key to be queried from the inactive memory table, a query result including the query value is obtained. In response to not finding a query value matching the key to be queried from the inactive memory table, the computing device may determine, based on the value to be queried, whether a query value matching the key to be queried is found from at least one first file stored in the first-level external memory. In response to finding a query value matching the key to be queried from the at least one first file, a query result including the query value is obtained.

[0170] In response to not finding a query value matching the key to be queried from at least one file, a determination is made based on the key to be queried whether a query value matching the key to be queried is found from an nth external storage file corresponding to the volume identifier to be queried and stored in the nth level of external storage. In response to finding a query value matching the key to be queried from the nth external storage file corresponding to the volume identifier to be queried and stored in the nth level of external storage, a query result including the query value is obtained. In response to not finding a query value matching the key to be queried from the nth external storage file corresponding to the volume identifier to be queried and stored in the nth level of external storage, n may be increased by 1, and the determination based on the key to be queried whether a query value matching the key to be queried is found from the nth external storage file corresponding to the volume identifier to be queried and stored in the nth level of external storage is repeated until a query value matching the key to be queried is found or a query value matching the key to be queried is not found from the last level of external storage.

[0171] According to an embodiment of the present application, since the nth file adopts a volume storage method for data storage, it is possible to query the query value matching the key to be queried from the nth external storage file corresponding to the volume identifier to be queried stored in the nth level external storage based on the key to be queried, without the need to query the nth file with other volume identifiers. Therefore, the query accuracy is improved, the query efficiency is improved, and the input and output overhead is reduced.

[0172] The above describes the data processing method and data query method provided by the embodiments of the present application.

[0173] Based on the above content, the data processing method and data query method provided in the embodiments of the present application are generally described below.

[0174] The embodiments of the present application optimize read performance while reducing storage overhead while achieving a higher write throughput through an intelligent layered design that combines a hybrid write layer with a separate storage layer. This can be reflected in the following aspects: Since data classification is not required in the hybrid layer, writing is faster, classification overhead is saved, and write throughput is improved. Since directional access to the volume data to be queried can be achieved in the separation layer, input and output overhead is reduced. Space amplification is reduced through dynamic splitting strategies, saving storage space. By adaptively adjusting the predetermined evaluation indicators that meet the target volume data based on load information, operation and maintenance are made more intelligent and suitable for large-scale production environments.

[0175] Regarding the target volume data splitting strategy, the first predetermined data volume and the predetermined data volume ratio are used to reduce the probability of storage fragmentation caused by premature splitting of highly mixed small data blocks. For example, the first predetermined data volume can be used to reduce the probability of small data blocks being split, and the predetermined data volume ratio can be used to reduce the probability of storage fragmentation. Regarding the retention strategy for other data, the amount of data stored in the first-tier external storage is reduced by retaining other data in the first-tier external storage and participating in subsequent rounds of merge operations when conditions are met.

[0176] Based on the same inventive concept as the aforementioned embodiment of the data processing method, the embodiment of the present application further provides a data processing device to implement the data processing method provided by the embodiment of the present application.

[0177] Figure 8 A block diagram of a data processing device according to an embodiment of the present application is shown.

[0178] like Figure 8 As shown, the apparatus 800 may include a first writing module 810 , an obtaining module 820 and a second writing module 830 .

[0179] The first writing module 810 is configured to write the data to be stored indicated by the write instruction to the active memory table stored in the memory, or write the data to be stored indicated by the flush instruction to the first file stored in the first-level external memory in response to the write instruction or the flush instruction.

[0180] The obtaining module 820 is configured to obtain target volume data having a target volume identifier among a plurality of volume identifiers in response to a merge operation between the (n-1)th level external storage and the nth level external storage.

[0181] The second writing module 830 is configured to write the target volume data having the target volume identifier to the nth file corresponding to the target volume identifier stored in the nth level external storage.

[0182] According to an embodiment of the present application, an active memory table may store multiple data items in a sequence of multiple keys. The data items may include keys with volume identifiers. The first file corresponding to the active memory table may include at least one data item with each of the multiple volume identifiers. n may be an integer greater than 1.

[0183] Based on the same inventive concept as the aforementioned data query method embodiment, the embodiment of the present application further provides a data query device to implement the data query method provided by the embodiment of the present application.

[0184] Figure 9 A block diagram of a data query device according to an embodiment of the present application is shown.

[0185] like Figure 9 As shown, the apparatus 900 may include an acquisition module 910 and a query module 920 .

[0186] The acquisition module 910 is configured to acquire a query instruction.

[0187] The query module 920 is configured to query for a query value matching the key to be queried from the nth external storage file corresponding to the volume identifier to be queried and stored in the nth external storage in response to the fact that a query value matching the key to be queried is not found in the active memory table and the inactive memory table stored in the memory, and in the at least one first file stored in the first-level external storage.

[0188] According to an embodiment of the present application, the query instruction may include a key to be queried having an identifier of the volume to be queried. n may be an integer greater than 1.

[0189] According to embodiments of the present application, any multiple modules among the first writing module 810, the obtaining module 820, and the second writing module 830 can be combined into a single module, any multiple modules among the obtaining module 910 and the query module 920 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the first writing module 810, the obtaining module 820, and the second writing module 830 can be at least partially implemented as a hardware circuit, and at least one of the obtaining module 910 and the query module 920 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or any other reasonable method of circuit integration or packaging, such as hardware or firmware, or any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the first writing module 810, the obtaining module 820 and the second writing module 830 can be at least partially implemented as a computer program module, and at least one of the obtaining module 910 and the query module 920 can be partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0190] Figure 10 A block diagram of an electronic device suitable for implementing a data processing method or a data query method according to an embodiment of the present application is shown.

[0191] like Figure 10 As shown, the electronic device 1000 according to an embodiment of the present application includes a processor 1001, which can perform various appropriate actions and processes according to the program stored in ROM 1002 (ROM is read-only memory) or the program loaded from the storage part 1008 to RAM 1003 (RAM is random access memory). The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0192] Various programs and data required for the operation of the electronic device 1000 are stored in the RAM 1003. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 1002 and / or the RAM 1003. It should be noted that the programs may also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0193] According to an embodiment of the present application, electronic device 1000 may further include an I / O interface 1005 (I / O stands for input / output), which is also connected to bus 1004. Electronic device 1000 may also include one or more of the following components connected to I / O interface 1005: an input unit 1006 including a keyboard, mouse, etc.; an output unit 1007 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage unit 1008 including a hard disk; and a communication unit 1009 including a network interface card such as a LAN card or modem. Communication unit 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1010 as needed, so that computer programs read from the removable media can be installed into storage unit 1008 as needed.

[0194] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0195] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0196] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the data processing method or data query method provided in the embodiments of the present application.

[0197] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the computer program is executed by the processor 1001. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0198] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0199] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0200] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0202] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0203] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A data processing method, characterized in that: include: In response to a write instruction or a flush instruction, writing the data to be stored indicated by the write instruction to an active memory table stored in the internal memory, or writing the data to be stored indicated by the flush instruction to a first file stored in the first-level external memory, wherein the active memory table stores multiple data in order of multiple keys, the data including a key with a volume identifier, the first file corresponding to the active memory table including at least one data with each of the multiple volume identifiers, the internal memory and the first-level external memory for storing the first file are located in a hybrid layer stored in a hybrid storage manner, the hybrid storage refers to not distinguishing volume identifiers and storing data according to global keys; and In response to a merge operation between the (n-1)th level external storage and the (n)th level external storage, target volume data having a target volume identifier among the plurality of volume identifiers is obtained, and the target volume data having the target volume identifier is written to an nth file corresponding to the target volume identifier and stored in the (n)th level external storage, wherein n is an integer greater than 1, and the second level external storage and levels of external storage above the second level external storage are located in a separate layer stored in a volume-based storage manner, wherein the volume-based storage refers to distinguishing volume identifiers and storing data according to local keys; The step of obtaining target volume data having a target volume identifier among the plurality of volume identifiers in response to the merge operation between the (n-1)th level external storage and the (n)th level external storage includes: In response to a merge operation between the first-tier external storage and the second-tier external storage, obtaining at least one data having each of the plurality of volume identifiers from at least one of the first files to obtain volume data having each of the plurality of volume identifiers; and In response to the plurality of volume data including volume data that meets at least one predetermined evaluation indicator, the volume data that meets the at least one predetermined evaluation indicator is used as the target volume data, or the target volume data is determined from the plurality of volume data using a pre-trained classification model based on the plurality of volume data.

2. The method according to claim 1, characterized in that In response to the plurality of volume data including volume data that satisfies at least one predetermined evaluation index, taking the volume data that satisfies the at least one predetermined evaluation index as the target volume data, comprising: Evaluating the plurality of volume data according to the at least one predetermined evaluation indicator to obtain evaluation results for each of the plurality of volume data, wherein the evaluation results indicate whether the volume data meets the at least one predetermined evaluation indicator; and In response to the respective evaluation results of the plurality of volume data including an evaluation result indicating that the volume data satisfies the at least one predetermined evaluation indicator, the volume data is used as the target volume data.

3. The method according to claim 2, characterized in that The at least one predetermined evaluation indicator includes at least one of the following: a first predetermined data volume, a predetermined data volume ratio, or a predetermined access frequency; The evaluation result of the target volume data indicates at least one of the following: the data volume of the target volume data is greater than or equal to the first predetermined data volume, the data volume ratio of the target volume data is greater than or equal to the predetermined data volume ratio, or the access frequency of the target volume data within a predetermined time period is greater than or equal to the predetermined access frequency, and the data volume ratio of the target volume data is the ratio between the data volume of the target volume data and the total data volume of multiple volume data.

4. The method according to claim 3, characterized in that The method further comprises: Adjust at least one of the first predetermined data volume, the predetermined data volume ratio, or the predetermined access frequency based on at least one of load information or inter-volume data distribution entropy, wherein the load information includes at least one of the following: the data volume stored in the first-level external memory, the first resource utilization rate, or the first input / output delay; and the inter-volume data distribution entropy indicates the degree of uniformity of distribution of the data in the multiple volumes.

5. The method according to claim 4, characterized in that The adjusting at least one of the first predetermined data volume, the predetermined data volume ratio, or the predetermined access frequency according to at least one of load information or inter-volume data distribution entropy includes: In response to at least one of the amount of data stored in the first-tier external memory being greater than or equal to a second predetermined amount of data, the inter-volume data distribution entropy being greater than or equal to a predetermined distribution entropy, the first resource utilization being greater than or equal to a predetermined utilization, or the first input / output delay being greater than or equal to a predetermined delay, reducing at least one of the first predetermined amount of data, the proportion of the predetermined amount of data, or the predetermined access frequency; and / or In response to at least one of the following: the amount of data stored in the first-level external memory is less than the second predetermined data amount, the inter-volume data distribution entropy is less than the predetermined distribution entropy, the first resource utilization is less than the predetermined utilization, or the first input-output delay is less than the predetermined delay, at least one of the first predetermined data amount, the proportion of the predetermined data amount, or the predetermined access frequency is increased.

6. The method according to claim 1, characterized in that The determining the target volume data from the plurality of volume data using a pre-trained classification model according to the plurality of volume data includes: Obtaining volume characteristic data of each of the plurality of volume data according to the plurality of volume data and the system data, wherein the volume characteristic data includes at least one of the following: a data volume of the volume data, a data volume ratio of the volume data, a second resource utilization rate, or a second input / output delay, wherein the data volume ratio of the volume data is a ratio of the data volume of the volume data to the total data volume of the plurality of volume data; Inputting the plurality of volume feature data into the pre-trained classification model respectively to obtain classification results of the plurality of volume data, wherein the classification results indicate whether the volume data is the target volume data; and In response to the classification results of each of the plurality of volume data including a classification result indicating that the volume data is target volume data, the volume data is set as the target volume data.

7. The method according to claim 6, characterized in that The pre-trained classification model is obtained by training an initial classification model using sample volume feature data of each of a plurality of sample volume data, wherein the sample volume feature data includes at least one of the following: the amount of data included in the sample volume data, the data volume ratio of the sample volume data, a third resource utilization rate, or a third input / output delay, wherein the data volume ratio of the sample volume data is the ratio between the data volume of the sample volume data and the total data volume of the plurality of sample volume data.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: In response to a merge operation on other data in at least one of the first files except the target volume data, obtaining merged data; and The merged data is written to a target first file stored in the first-level external memory, wherein the target first file is a first file in at least one of the first files or the target first file is a newly created file other than at least one of the first files.

9. The method according to claim 8, characterized in that The method further comprises: A predetermined identifier associated with the target first file is configured, wherein the predetermined identifier indicates that the target first file has participated in at least one of a round of a merge operation or a merge source, and does not participate in a predetermined number of merge operations after the round.

10. The method according to claim 9, characterized in that The method further comprises: In response to a similar first file having a similarity between the first-level external memory storage and the target first file being greater than or equal to a predetermined similarity, a merge operation is performed between the target first file and the similar first file when a predetermined identifier associated with the target first file indicates not to participate in the merge operation.

11. The method according to any one of claims 1 to 7, wherein In response to a disk flushing instruction, writing the to-be-stored data indicated by the disk flushing instruction into a first file stored in a first-level external memory includes: Responding to the disk flushing instruction, using a disk flushing thread to obtain the to-be-stored data indicated by the disk flushing instruction from an inactive memory table stored in the memory, wherein the inactive memory table has the same storage data and storage method as the active memory table; and The disk flushing thread is used to write the data to be stored indicated by the disk flushing instruction into the first file stored in the first-level external memory.

12. The method according to claim 11, characterized in that The method further comprises: In response to the data amount stored in the active memory table being equal to the third predetermined data amount, the write thread is used to configure the status flag of the active memory table as a read-only flag to obtain the inactive memory table.

13. The method according to claim 11, characterized in that The method further comprises: Using a write thread to store the inactive memory table in a dynamic array; and In response to completing the disk flushing operation on the inactive memory table, the inactive memory table is removed from the dynamic array by the disk flushing thread.

14. The method according to any one of claims 1 to 7, wherein In response to a write instruction, writing the to-be-stored data indicated by the write instruction into an active memory table stored in a memory includes: Obtaining the write instruction, wherein the write instruction includes the data to be stored, and the data to be stored includes a value to be stored and a key to be stored having an identifier of a volume to be stored; Determining a global to-be-written location from an active memory table of the memory according to the to-be-written key; and The data to be stored indicated by the write instruction is written into the global write location.

15. The method according to claim 14, characterized in that Writing the data to be stored into the global to-be-written location includes: The data to be stored is written to the global write location using an atomic operation.

16. The method according to any one of claims 1 to 7, characterized in that The step of obtaining target volume data having a target volume identifier among the plurality of volume identifiers in response to a merge operation between the (n-1)th level external storage and the (n)th level external storage includes: In response to a merge operation between the (n-1)th level external storage and the (n)th level external storage, obtaining (n-1)th target volume data having a target volume identifier among the plurality of volume identifiers from at least one (n-1)th file and obtaining (n)th target volume data having the target volume identifier from at least one (n)th file, the (n-1)th file being stored in the (n-1)th level external storage, the (n)th file being stored in the (n)th level external storage, and n being an integer greater than 2; and The target volume data having the target volume identifier is obtained according to the (n-1)th target volume data and the (n)th target volume data having the target volume identifier.

17. The method according to any one of claims 1 to 7, characterized in that Also includes: At least one of a storage medium or a compression strategy adapted to the nth file is configured, wherein the file identifier of the nth file has the target volume identifier.

18. A data query method, characterized in that: Applied to data obtained by the data processing method according to any one of claims 1 to 17, the data query method comprises: Obtaining a query instruction, wherein the query instruction includes a to-be-queried key having an identifier of a to-be-queried volume; and In response to the fact that a query value matching the key to be queried is not found from the active memory table and the inactive memory table stored in the memory, and at least one first file stored in the first-level external memory, a query value matching the key to be queried is queried from the nth external memory file corresponding to the volume identifier to be queried and stored in the nth-level external memory, where n is an integer greater than 1.

19. A data processing device, characterized in that: include: a first writing module configured to, in response to a write instruction or a flush instruction, write the to-be-stored data indicated by the write instruction to an active memory table stored in the internal memory, or write the to-be-stored data indicated by the flush instruction to a first file stored in the first-tier external memory, wherein the active memory table stores a plurality of data in an ordered order of a plurality of keys, the data including a key having a volume identifier, the first file corresponding to the active memory table including at least one data having each of the plurality of volume identifiers, the internal memory and the first-tier external memory for storing the first file being located in a hybrid layer stored in a hybrid storage manner, the hybrid storage not distinguishing between volume identifiers and storing data according to a global key; and an obtaining module configured to obtain target volume data having a target volume identifier among the plurality of volume identifiers in response to a merge operation between the (n-1)th level external storage and the (n)th level external storage; and a second writing module configured to write the target volume data having the target volume identifier to an nth file corresponding to the target volume identifier stored in the (n)th level external storage, wherein n is an integer greater than 1, and the second level external storage and levels of external storage above the second level external storage are located in a separate layer stored in a volume-based storage manner, wherein the volume-based storage refers to distinguishing volume identifiers and storing data according to local keys; Wherein, the obtaining module is further configured to: In response to a merge operation between the first-tier external storage and the second-tier external storage, obtaining at least one data having each of the plurality of volume identifiers from at least one of the first files to obtain volume data having each of the plurality of volume identifiers; and In response to the plurality of volume data including volume data that meets at least one predetermined evaluation indicator, the volume data that meets the at least one predetermined evaluation indicator is used as the target volume data, or the target volume data is determined from the plurality of volume data using a pre-trained classification model based on the plurality of volume data.

20. A data query device, characterized in that: Applied to data obtained by the data processing device according to claim 19, the data query device comprises: an acquisition module configured to acquire a query instruction, wherein the query instruction includes a to-be-queried key having an identifier of a to-be-queried volume; and The query module is configured to query for a query value matching the key to be queried from an nth external memory file corresponding to the volume identifier to be queried and stored in an nth level external memory in response to the fact that a query value matching the key to be queried is not found from the active memory table and the inactive memory table stored in the memory, and from at least one first file stored in the first-level external memory, where n is an integer greater than 1.

21. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 17 or the steps of the method according to claim 18.

22. A computer-readable storage medium, characterized in that A computer program or instruction is stored thereon, wherein when the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 17 or the steps of the method according to claim 18 are implemented.

23. A computer program product, characterized in that The method comprises a computer program or instructions, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17 or the steps of the method according to claim 18.

Images