Data system, method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides a data system, method and device, equipment, a medium and a program product. The data system comprises a data node, and the data node comprises at least one data partition; wherein any data partition in the at least one data partition is divided into a plurality of sub-data partitions; each sub-data partition in the plurality of sub-data partitions in any data partition is loaded with a storage engine. By adopting the system provided by the embodiment of the invention, the engine performance of the storage engine in the database can be effectively improved, for example, the query efficiency of the storage engine is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of databases, and specifically to a data system, a data processing method, a data processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] A data system can generally refer to a system used to store massive amounts of data; a data system can implement data operations on the data in the data system based on a storage engine.

[0003] Currently, data systems can store data in partitions, with each partition hosting its own storage engine. This mapping between storage engines and data partitions allows a single storage engine to be responsible for data operations within a single partition. However, a single data partition in a data system often contains a large amount of data, which severely degrades the performance of the storage engine. Summary of the Invention

[0004] Embodiments of the present application provide a data system, method, apparatus, device, medium, and program product that can effectively improve the engine performance of a storage engine.

[0005] In one aspect, an embodiment of the present application provides a data system, wherein the data system includes a data node, and the data node includes at least one data partition; wherein:

[0006] Any data partition in at least one data partition is divided into a plurality of sub-data partitions; the data capacity of each of the plurality of sub-data partitions in any data partition is smaller than the data capacity of any data partition; the data partition is a data migration unit of the data system;

[0007] Each of the multiple sub-data partitions in any data partition is loaded with a storage engine; the storage engine loaded in any sub-data partition is used to perform data operations on any sub-data partition.

[0008] In one implementation, the data system is a distributed data system, which includes multiple clusters, each of which includes data nodes; wherein,

[0009] The multiple clusters include a first cluster and a second cluster; when data migration is required between the first cluster and the second cluster, data migration is performed using data partitions as migration units; or,

[0010] The cluster includes a first data node and a second data node. When data migration is required between the first data node and the second data node, the data migration is performed using a data partition as a data migration unit.

[0011] In one implementation, the data system includes an access node; the client communicates data with a data node in the data system via the access node;

[0012] The access node communicates data with the data partitions in the data node;

[0013] The data partition communicates data with the sub-data partitions belonging to the data partition.

[0014] In one implementation, the data capacity of any data partition is equal to the sum of the data capacities of the sub-data partitions in the plurality of sub-data partitions in the data partition;

[0015] The partitioning strategy used when a data partition is divided into multiple sub-data partitions includes at least one of the following:

[0016] Classify according to the business attributes of the business to which the data belongs; or

[0017] Divide by data attributes; or,

[0018] Divide by the hash value of the data.

[0019] In one implementation, any sub-data partition is represented as a target sub-data partition, and the storage engine loaded in the target sub-data partition is represented as a reference storage engine; the reference storage engine includes: a first part and a second part; the first part includes multiple memory data structures, and the second part is a multi-level data structure, and each level in the second part includes one or more ordered data structures;

[0020] Among them, if the number of memory data structures in the reference storage engine is the same as the number of memory data structures in the storage engine loaded before the data partition is split, and the number of ordered data structures of each level in the second part of the reference storage engine is the same as the number of ordered data structures of the corresponding level in the storage engine loaded before the data partition is split, then the data volume of the memory data structure in the reference storage engine is smaller than the data volume of the memory data structure in the storage engine loaded before the data partition is split, and the data volume of the ordered data structures of each level in the second part of the reference storage engine is smaller than the data volume of the ordered data structures of the corresponding level in the storage engine loaded before the data partition is split.

[0021] In one implementation, the number of ordered data structures at any level in the second portion of the reference storage engine is N, where N is an integer greater than 1;

[0022] N ordered data structures are split into M ordered data structures; M is an integer greater than N;

[0023] The data volume of any one of the M ordered data structures is smaller than the data volume of any one of the N ordered data structures.

[0024] In one implementation, each data partition in at least one data partition is divided into a plurality of reference data partitions; the data capacity of the reference data partition is smaller than the data capacity of the data partition to which the reference data partition belongs;

[0025] Each reference data partition is divided into multiple sub-data partitions.

[0026] In an embodiment of the present application, it is supported to define multiple sub-data partitions in each data partition included in the data node in the data system; that is, it is supported to divide or split sub-data partitions within the data partition, and the amount of data that can be stored in the sub-data partition is smaller than the amount of data that can be stored in the complete data partition. Furthermore, a storage engine is loaded separately for each sub-data partition. Compared with loading a storage engine for a complete data partition in the related art, it can ensure that the storage engine only needs to manage the data contained in the sub-data partition with a smaller data volume, thereby optimizing the engine performance of the storage engine. Furthermore, the divided data system still uses the data partition as the data migration unit, which not only reduces the workload of the storage engine, but also does not increase the management scale of the data system.

[0027] On the other hand, an embodiment of the present application provides a data processing method, which is applied to the aforementioned data system, and includes:

[0028] Obtaining a data operation request, where the data operation request is used to request execution of a data operation on target data in a data system;

[0029] Performing first-level routing processing on the data system according to the data operation indicated by the data operation request, routing from the data system to a target data partition associated with the target data;

[0030] Performing a second level routing process on the target data partition, routing from the target data partition to a reference sub-data partition associated with the target data;

[0031] Data operations are performed on target data in the reference sub-data partition through the storage engine loaded in the reference sub-data partition.

[0032] In another aspect, an embodiment of the present application provides a data processing device, which is installed in the aforementioned data system; the device includes:

[0033] an acquisition unit, configured to acquire a data operation request, wherein the data operation request is used to request execution of a data operation on target data in the data system;

[0034] a processing unit, configured to perform a first level routing process on the data system according to the data operation indicated by the data operation request, and route the data from the data system to a target data partition associated with the target data;

[0035] The processing unit is further configured to perform a second level routing process on the target data partition, routing from the target data partition to a reference sub-data partition associated with the target data;

[0036] The processing unit is further configured to perform data operations on target data in the reference sub-data partition through a storage engine loaded in the reference sub-data partition.

[0037] In one implementation, the data node caches an attribute list, the attribute list including one or more fields, the fields being obtained based on metadata of the data stored in the data node;

[0038] The processing unit is configured to perform first-level routing processing on the data system according to the data operation indicated by the data operation request, and when routing from the data system to the target data partition associated with the target data, specifically to:

[0039] Determine a first field from the attribute list; the first field is used to indicate a routing path of a target data partition associated with the target data from at least one data partition in the data node;

[0040] According to the routing path indicated by the first field, routing is performed from at least one data partition in the data node to the target data partition.

[0041] In one implementation, the processing unit is configured to perform the second-level routing processing on the target data partition, and when routing from the target data partition to the reference sub-data partition related to the target data, is specifically configured to:

[0042] determining a second field from the attribute list, the second field being used to indicate a routing path for determining a reference sub-data partition related to the target data from the target data partition;

[0043] According to the routing path indicated by the second field, routing is performed from the multiple sub-data partitions in the target data partition to the reference sub-data partition.

[0044] In an embodiment of the present application, when the data partition of the data node in the data system is divided into sub-data partitions, when any data node receives a data operation request related to the target data sent from the outside, it can perform a first-level routing processing on the data system in response to the data operation request, aiming to find the target data partition from the data system, and the target data partition has an associated relationship with the target data (such as in a write data scenario, it is hoped that the target data will be written to the target data partition, and in a read data scenario, the target data belongs to the target data partition, etc.). Then, a second-level routing processing is performed on the target data partition to find a reference sub-data partition from the target data partition, and the reference sub-data partition has an associated relationship with the target data (similar to the above explanation of the associated relationship); since the second-level routing processing is performed in a single data partition, the routing cost is very low and will not cause problems such as waste of resources. In this way, data operations can be performed on the target data in the reference sub-data partition through the storage nodes loaded in the reference sub-data partition. It can be seen from this that since the corresponding sub-data partition contains less data, the multi-layer ordered data structure contained in the storage engine, while the number of ordered data structures in each layer remains unchanged, the data contained in a single ordered data structure is also reduced. For the storage engine, performing data operations in the corresponding sub-data partition is compared to performing data operations in the complete data partition. This effectively reduces the amount of data that the storage engine needs to process when performing data operations, thereby improving the working performance of the storage engine while ensuring the correctness of data operations.

[0045] In another aspect, an embodiment of the present application provides a computer device, comprising:

[0046] a processor for loading and executing computer programs;

[0047] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method is implemented.

[0048] On the other hand, the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor and executing the above-mentioned data processing method.

[0049] In another aspect, the present application provides a computer program product or computer program, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-mentioned data processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1a It is a schematic diagram of the architecture of a distributed database;

[0052] Figure 1b This is a structural diagram of an LSM storage engine;

[0053] Figure 1c This is a schematic diagram of the architecture of a distributed database equipped with an LSM storage engine.

[0054] Figure 2 This is a schematic diagram of the architecture of a data system provided by an exemplary embodiment of the present application;

[0055] Figure 3a This is a schematic diagram of dividing data partitions in a data system provided by an exemplary embodiment of the present application;

[0056] Figure 3b is a schematic diagram of another method of dividing data partitions in a data system provided by an exemplary embodiment of the present application;

[0057] Figure 4 is a flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0058] Figure 5 This is a schematic diagram of the structure of a data system after data partitioning provided by an exemplary embodiment of the present application;

[0059] Figure 6 is a flowchart of another data processing method provided by an exemplary embodiment of the present application;

[0060] Figure 7 This is a flow chart of performing secondary routing in a data system provided by an exemplary embodiment of the present application;

[0061] Figure 8 It is a structural diagram of an existing reduced partiton;

[0062] Figure 9 This is a schematic diagram of the structure of an existing reduced SSTable;

[0063] Figure 10This is a schematic diagram of splitting a data partition and reducing the size of an ordered data structure in the split sub-data partitions provided by an exemplary embodiment of the present application;

[0064] Figure 11 This is a schematic diagram of splitting a data partition after reducing its size, provided by an exemplary embodiment of the present application;

[0065] Figure 12 This is a schematic diagram of an exemplary embodiment of the present application, which provides a method of splitting a data partition after reducing its size, and reducing an ordered data structure in the split sub-data partitions;

[0066] Figure 13 is a structural diagram of a data processing device provided by an exemplary embodiment of the present application;

[0067] Figure 14 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0069] The following first introduces the technical terms and related concepts involved in the embodiments of this application, among which:

[0070] 1. Data system

[0071] A data system is a system that uses data as its object and implements unified management and organization of data. Depending on the different data management methods, a data system can be simply understood as a system consisting of data acquisition, data storage, data updating, data circulation, and data mining. For example, a data system may include a database system, which is a system consisting of a database and its management software, and the management software in the system can implement data management and organization in the database. For another example, a data system may include a data mining system, which is a system mainly used to mine specified data from a database (or data warehouse, information library) that stores data. The embodiments of the present application mainly relate to database systems in data systems, and specifically to databases in database systems. Among them, the database in the database system is a warehouse that organizes, stores, and manages data according to data structure; that is, the database can be understood as a collection of large amounts of data that is stored in a computer for a long time, organized, shareable, and uniformly managed, so the database can also be called a storage system. In other words, the database can be regarded as an electronic filing cabinet - a place to store electronic files, and users can perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data that is stored in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0072] To adapt to the increase in data volume brought about by the development of the Internet, distributed databases have been developed. A distributed database can refer to a group of logically interconnected databases distributed on a computer network. It emphasizes storing data in databases at different physical locations to solve the computing and storage bottlenecks brought about by stand-alone databases. An exemplary distributed database architecture diagram can be found in Figure 1a ;exist Figure 1aThe distributed database shown includes: a client 101, a metadata server set, and a data server set. The metadata server set includes one or more metadata servers 102, and the data server set includes one or more data servers 103. Specifically, the following: ① The client is responsible for reading and writing data in the distributed database, caching metadata and file data, etc. The client can specifically refer to a terminal used by a user, which can include but is not limited to: (Mobile Internet Devices, MIDs), smart TVs, in-car devices, head-mounted devices, game controllers, remote controls, arcade equipment, and other smart devices. Users use this client to access the database through the server to perform data operations such as reading and writing data in the database. ② The metadata server is responsible for managing the metadata of each data in the distributed database and processing requests from clients. It is the core component of the entire distributed database. The metadata of the data can be called relay data, intermediary data, etc. It is information used to describe the data attributes and is used to support functions such as indicating the data storage location, historical data, resource search, and file records. In other words, this metadata is similar to an electronic directory of data, which can achieve the purpose of facilitating data retrieval. ③ The data server is the device responsible for storing data and can ensure the availability and integrity of the data. Among them, the servers mentioned above (such as metadata servers or data servers) can be independent physical servers, or server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in this application.

[0073] It should be understood that Figure 1aThis is only an exemplary architectural diagram of a distributed database provided for the embodiments of the present application; in actual applications, the architecture of the distributed database may also change. For example, the architecture of the distributed database may also include one or more database clusters (referred to as clusters for short), and a cluster is the concept of combining multiple data servers together to form an overall system. Among them, the data servers in the cluster can be represented as data nodes (referred to as nodes for short, specifically computer devices (such as servers)), each data node can run independently and has its own computing and storage resources. Under this architecture, there is a corresponding relationship between the metadata server and the cluster, specifically a metadata server (the number of metadata servers can also be multiple) is responsible for managing the metadata of the data stored in each data node in the same cluster, and processing client requests for the same cluster.

[0074] Furthermore, a data node in a cluster can include at least one data partition (represented as a partition, region, tablet, vnode, vBucket, or shard). This data partition can be understood as the storage space within the data node used to store data. A data node can store data in different partitions. When receiving a request for data, it can route it to the data partition associated with the data (e.g., the data belongs to a specific data partition). This allows operations such as data traversal to be performed only within that data partition, significantly improving data operation efficiency and avoiding resource waste.

[0075] 2. Storage Engine

[0076] Storage engines may include, but are not limited to, hash storage engines, B (InnoDB) tree storage engines, and LSM (logstructured merge trees) storage engines. Among them: (1) The hash storage engine maintains a hash table, which maps a unique index of each data in the database, so that data operations (such as data query, data addition, deletion, and modification) on the data in the database can be implemented through the hash table. (2) The B-tree storage engine relies on pages to organize the data in the database. Each page corresponds to a leaf node in a B-tree (a self-balancing tree structure that can keep data in order). The leaf node is used to store the data in the database, while the non-leaf nodes in the B-tree are used to store index information for the database.

[0077] (3) The LSM storage engine as a whole can be composed of two parts: memory and disk in a computer device (i.e., the aforementioned data server or data node). The LSM storage engine is referred to as a reference storage engine. The exemplary structure of the reference storage engine can be found in Figure 1b ,exist Figure 1b The reference storage engine shown includes: a first part and a second part; wherein the first part is the memory part, which includes a writable (i.e., supports writing data) MemTable (memory data structure, or MemTable file). The second part is the disk part, which is divided into multiple layers (Level), that is, the second part is a multi-level data structure, and each level in the second part includes one or more ordered data structures (Sorted StringTable, SSTable), which is also called an SSTable file, or an ordered key-value pair set, containing some data files organized in order, and supports speeding up data queries in SSTable files by establishing key indexes and Bloom filters. The first and second parts of the reference storage engine (i.e., the LSM storage engine) are introduced below, wherein:

[0078] ① The memory data structure MemTable in the first part can be used to store data that needs to be written to the second part (specifically, the ordered data structure SSTable written to the second part), and new data written to the data partition is always inserted into the memory data structure MemTable first. The memory data structure can serve both read data services and write data services. For example, when reading data stored in the data partition externally, that is, before reading the SSTable file, always query the MemTable in the first part first; if the data you want to read can be queried from the MemTable, there is no need to query the SSTable file in the second part. For another example, when writing new data to the data partition, the new data is written to the Mem Table file in the first part; when a MemTable file in the first part is full (that is, the data capacity of the MemTable file (that is, the maximum amount of data that the MemTable file can contain) is fully occupied, or the amount of data that the Mem Table file can carry reaches the maximum), the MemTable file changes from a writable state to a non-writable state, and a MemTable file is added to the first part to replace the full MemTable file (such as Figure 1b As shown, the number of MemTable files in the first part is always maintained at 6, and the value 6 here is an example), and the background thread writes the content of the MemTable file to a SSTable file in the second part.

[0079] ②The second part includes multi-level data structures. Figure 1bThe second part of the reference memory engine shown includes three levels. Among them: the number of ordered data structures SSTable included in the first level Level 0 is the same as the number of memory data structures MemTable included in the first part, and the Level 0 level contains SSTable files converted from the MemTable files in the first part. Considering that the same data may exist in different MemTable files, there may be overlapping data between the SSTable files in the Level 0 level (such as data stored in the form of key-value pairs); when the number of SSTables in the Level 0 level reaches a certain threshold (such as the threshold is 6, that is, it is necessary to maintain a certain number of SSTable files contained in each level), the database (such as LevelDB (an open source key-value storage system)) will trigger a merge operation (Compaction) to deduplicate the data of the SSTable files in the Level 0 level and merge them into the Level 1 level. Similarly, the number of ordered data structures included in the second level, Level 1, is smaller than the number of ordered data structures included in Level 0; and the SSTable files in Level 1 contain SSTable files merged from Level 0. At this time, due to incomplete deduplication when merging from Level 0 or other factors, there may be slight overlap between the SSTables in Level 1; when the number of SSTable files in Level 1 reaches a certain threshold (such as the threshold is 3), the database will trigger a merge operation to merge the SSTable files in Level 1 into Level 2. Similarly, the number of ordered data structures included in the third level, Level 2, is smaller than the number of ordered data structures included in Level 2, and starting from Level 2, the probability of overlap between SSTable files in each level becomes smaller and smaller (or even no overlap exists), which means that each key-value pair will only exist in one SSTable file.

[0080] It should be noted that, as mentioned above, the database supports the organization and storage of data, while the storage engine is the software or system responsible for the management and processing of data in the database. In short, the database uses the storage engine to implement data operations such as retrieving data, inserting data (or adding new data), updating data, and deleting data. It has been found in practice that the engine performance of the storage engine loaded in the data partition is poor due to the limitation of the data capacity of the data partition (i.e., the maximum amount of data that can be carried) of the data partition in the database. Below, taking the data system as a distributed database and the storage engine loaded in the distributed database (the storage engine is specifically loaded in each data partition of the data node in the distributed database system) as an LSM storage engine as an example, the reason why the engine performance of the LSM storage engine is affected by the data capacity of the data partition is explained.

[0081] An exemplary architecture diagram of a distributed database with an LSM storage engine can be found in Figure 1c ,exist Figure 1c The distributed database shown includes clusters, etc. Different clusters are composed of different data nodes, such as Figure 1cThe cluster in is composed of data node 104 and data node 105. Among them, data node 104 includes two data partitions, namely data partition 1 and data partition 2; the data partition 1 and data partition 2 can be used to realize the decentralized storage and management of the data in data node 104, and data partition 1 and data partition 2 are respectively loaded with LSM storage engines. For databases (or storage systems) based on LSM indexes, the reading and writing of the ordered data structure SSTable in the LSM storage engine loaded in each data partition is a key bottleneck. It has been found in practice that the current LSM storage engine has write amplification, read amplification and locking problems. For example, as the amount of data in the data partition increases, the write amplification, read amplification and locking problems of the LSM storage engine are greatly aggravated. Among them: ① Write amplification: When the number of low-level SSTable files reaches a threshold (or when the first part of the in-memory data structure in the LSM, MemTable, is full), the low-level SSTables can be merged and compacted to merge to a higher level. However, during merging and compacting, duplicate data may exist in different SSTable files, causing duplicate data to be rewritten multiple times. This phenomenon is called write amplification. ② Read amplification: When an external system or client queries data, the queried data may be distributed across multiple SSTable files, requiring searches across multiple SSTable files. This phenomenon is called read amplification. ③ Lock contention: In a multi-threaded environment, LSM indexes require locks to protect the integrity of the data structure (i.e., locks are used to lock the data to be processed). Due to the large data capacity of data partitions, the range of data that needs to be locked during merging and compaction is also larger. Furthermore, when the lock is released, multiple threads will compete for the same lock. This lock contention and concurrency control mechanism can lead to performance degradation of the storage engine. It should be noted that excessively large data partitions can also cause other problems. For example, since different SSTables may contain duplicate data and / or some temporary intermediate state SSTables may be generated during the SSTable merging and compression process, the disk space is enlarged, resulting in a waste of disk space and requiring higher dimensionality costs; querying the same data in multiple levels leads to low data query efficiency; at the same time, the merging / compression of SSTables in the background will also compete with the foreground query for new data writing for system resources (such as CPU (Central Processing Unit), memory, disk and I / O (Input / Output)), causing resource competition and passive merging of data.

[0082] In order to improve the engine performance of the storage engine in a distributed database, an embodiment of the present application provides an optimization solution for the storage engine in a distributed database, and specifically provides a new data system and a data processing solution based on the data system. Among them, the optimization solution provided by the embodiment of the present application can be applied to any distributed data system that uses a storage engine, and the embodiment of the present application does not limit the application scenario of the optimization solution. The optimization solution provided by the embodiment of the present application, on the one hand, supports further segmentation of the data partitions in the data nodes to obtain a new data system; on the other hand, a data processing solution for the data system is given based on the new data system; the optimization solution can improve and optimize the engine capabilities of the storage engine without bringing new problems to the storage engine. The embodiment of the present application takes the storage engine as LSM as an example, which is specially explained here.

[0083] Specifically, the optimization solution provided in the embodiments of the present application supports defining multiple sub-data partitions (or mini partitions) within a data partition within a data node, similar to using multiple sub-data partitions to disperse the data originally stored in a single data partition. Furthermore, a storage engine is provided for each of the multiple sub-data partitions, so that the storage engine loaded into the sub-data partition is responsible for data storage and management within the sub-data partition. Considering that the amount of data managed by the storage engine loaded into the sub-data partition is much smaller than the amount of data on the data node, the embodiments of the present application support representing the LSM storage engine loaded into the sub-data partition as a mini LSM.

[0084] The data system proposed in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0085] Figure 2The schematic diagram of the architecture of a data system provided by an exemplary embodiment of the present application is shown; the data system includes a data node 201, and the data node 201 may include at least one data partition (such as data partition 202). In the case where the data system can be a distributed data system, the data system may include multiple clusters, and different clusters may be responsible for data of different business attributes; a single cluster may include one or more data nodes. Furthermore, the embodiment of the present application supports the division of any data partition in at least one data partition so that any data partition in at least one data partition is divided into multiple sub-data partitions, wherein the data capacity of each sub-data partition in the multiple sub-data partitions in any data partition (i.e., the maximum amount of data that can be stored) is less than the data capacity of any data partition; for example, data partition 202 is divided into sub-data partition 2021 and sub-data partition 2022, and the data capacity of sub-data partition 2021 and sub-data partition 2022 is less than the data capacity of data partition 202. Furthermore, a storage engine is loaded for each of the multiple sub-data partitions in any data partition that has been divided, that is, each of the multiple sub-data partitions in any data partition is loaded with a storage engine. At this time, the storage engine loaded in any sub-data partition is used to perform data operations on any sub-data partition.

[0086] Based on the above Figure 2 Regarding the framework of the data system provided in the embodiments of the present application, it is necessary to explain that:

[0087] (1) In the case where a data system includes multiple data nodes, or a data node includes multiple data partitions, the embodiment of the present application supports the division of each data partition in all data nodes into sub-data partitions and the loading of storage engines related to the sub-data partitions; if the data system includes two data nodes, and each data node includes two data partitions, then the embodiment of the present application supports the division of each data partition in the two data nodes into sub-data partitions and the loading of storage engines. Figure 3a As shown, assuming that a cluster of a data system includes data node 1 and data node 2, and data node 1 and data node 2 respectively contain two data partitions, then the two data partitions in data node 1 and the two data partitions in data node 2 are supported to be divided into sub-data partitions to improve the engine performance of the storage engine in each data partition.

[0088] In addition, the embodiment of the present application also supports the dynamic division of some data partitions according to the resource occupancy or deployment of the data system to meet the needs of the system administrator to dynamically manage the data system. Figure 3bAs shown, assuming that the data system includes data node 1 and data node 2, data node 1 is used to store data belonging to business type 1, and data node 2 is used to store data belonging to business type 2, and the amount of data belonging to business type 1 is much larger than the amount of data belonging to business type 2, then the system administrator can dynamically set the data partitions in data node 1 for division, while retaining the original settings of the data partitions in data node 2. While ensuring that data node 2 operates normally according to the original logic, the engine performance of the storage engine in each data partition in data node 2 is effectively improved.

[0089] (2) The present application embodiment supports the use of a partitioning strategy to partition a data partition to obtain multiple sub-data partitions corresponding to the data partition. The present application embodiment does not limit the specific content of the partitioning strategy; the partitioning strategy used when any data partition is divided into multiple sub-data partitions may include at least one of the following:

[0090] 1. Partitioning by the business attributes of the business to which the data belongs. Business attributes generally refer to data fields or columns related to the business logic or business rules applied to the data. They describe business characteristics or status and can be used to support business processes, decision-making, and reporting. For example, a product table contains business attributes, which may include information such as the product's unique identifier, product name, product price, and product creation date. In other words, partitioning by business attributes allows data belonging to the same business attributes to be stored in the same sub-data partition. This reduces the data capacity of the sub-data partition while also enabling categorized data storage, improving query speed and efficiency when querying data based on business attributes. 2. Partitioning by data attributes. Data attributes can refer to attributes used in database design to describe the data characteristics of an entity or object. These attributes store data values ​​associated with the entity / object instance. Compared to business attributes, data attributes focus more on the data actually stored and manipulated. That is to say, the embodiment of the present application supports partitioning according to the data attributes of the data, such as storing data of the same data type in the same sub-data partition, and storing data corresponding to frequently queried data attributes in the same sub-data partition, etc., which greatly improves the storage and query efficiency of the data. ③ Partitioning according to the hash value of the data; specifically including further dividing the data partition according to the hash value of the data, aiming to evenly distribute the data in the data partition to multiple sub-data partitions, achieve load balancing and improve query efficiency. Among them, when partitioning according to the hash value, a hash function is applied to a certain field of the data (such as a primary key or a unique identifier), and the hash result is mapped to a partition identifier of a sub-data partition (such as used to uniquely identify a sub-data partition); the partition identifier determines which sub-data partition the data should be stored in; since the hash function usually has the characteristic of uniform distribution, the division of the sub-data partitions according to the hash value can ensure the uniform distribution of data among the sub-data partitions in the data partition.

[0091] It should be understood that the partitioning strategy is intended to: split the data capacity of any data partition so that after the split, the data capacity of each of the multiple sub-data partitions in any data partition is smaller than the data capacity of any data partition, but the data capacity of any data partition is equal to the sum of the data capacity of each of the multiple sub-data partitions in any data partition; in this way, for the storage engine loaded in a single sub-data partition, it only needs to manage the data contained in the corresponding sub-data partition, and the amount of data required for data operations such as merging, compression, and reading and writing is greatly reduced, effectively improving the engine performance of the storage engine. Therefore, the above are only several exemplary partitioning strategies given in the embodiments of the present application. In actual applications, rich partitioning strategies can also be customized according to the management needs of the system administrator, which is specially explained here.

[0092] (3) Considering that after the data partition is divided in the embodiment of the present application, the multiple sub-data partitions obtained still belong to the corresponding data partition, that is, the multiple sub-data partitions are still managed by the corresponding data partition; therefore, when the data system provided by the embodiment of the present application performs data migration, the data migration is still performed with the data partition as the data migration granularity (or data migration unit); that is, the data partition in the data system is the data migration unit of the data system. Among them, data migration in the data system can be the process of converting and storing data between different data partitions. Optionally, when the data system does not include a cluster, the data system can realize the migration of data in a data partition in a data node to a data partition in another data node.

[0093] Optionally, if the data system is a distributed data system, which may include multiple clusters, each of which contains data nodes, data migration can be performed across clusters or across nodes within the same cluster. These two types of data migration are briefly described below. Cross-cluster migration refers to migrating data from one cluster to another. For example, when the data in a cluster continues to grow and the current cluster size cannot be directly modified to meet demand, a new cluster can be created and, through cluster migration, data can be quickly migrated to the new cluster, expanding data capacity while meeting business needs. In the case of a distributed data system, clusters still perform data migration at the data partition granularity. This allows the data capacity of individual sub-data partitions within a data partition to be reduced without increasing the cluster size, thereby minimizing the data migration burden. Cross-node migration refers to migrating data from one data node within a cluster to another data node within the same cluster. For example, when the data or load between different data nodes in the cluster is uneven, it may be necessary to perform data migration between different data nodes to achieve data balance on each data node. For another example, in order to achieve cluster expansion, it is necessary to add data nodes to the cluster. Then, through data migration between data nodes, the data in the original data nodes in the cluster can be moved to the newly added empty data nodes to share the data pressure of the original data nodes while making use of the new data nodes. For another example, after one or more data nodes in the cluster fail, it is necessary to disperse and migrate the data in the failed data nodes to other normally operating data nodes in the cluster to avoid the data in the failed data nodes being unable to be used normally, thereby maintaining the normal operation of the cluster.

[0094] In a specific implementation, it is assumed that a distributed data system includes multiple clusters, each cluster includes data nodes, and the multiple clusters include a first cluster and a second cluster, and the first cluster and the second cluster are different. Then when there is a need for data migration between the first cluster and the second cluster (that is, when data migration is required between the first cluster and the second cluster), data migration is performed between the first cluster and the second cluster using the data partitions in the data nodes contained in the cluster as the data migration granularity (or migration unit). For example, a data partition in a data node in the first cluster is migrated as a whole to the second cluster; during this migration process, multiple sub-data partitions contained in the data partition will be migrated from the first cluster to the second cluster. Similarly, when different data nodes in the same cluster have data migration needs, such as when data migration is required between the first data node and the second data node in the cluster, data migration is still performed using data partitions as the data migration unit. Based on this, since the complexity of cluster management is reflected in the mapping relationship between the cluster management data partitions and data nodes (this relationship is strongly related to the routing when reading and writing data), and the number of partitions that can be managed by the effective resources of the cluster is upper bounded, the embodiment of the present application divides the data partitions, and the multiple sub-data partitions still belong to the data partitions (that is, the data partitions are still used as the data migration units between clusters), which will not increase the number of data partitions in the data nodes in the cluster, and thus will not affect the scale and complexity of cluster management.

[0095] Based on the above introduction to the framework of the data system provided in the embodiment of the present application, the data processing solution based on the data system is introduced below in conjunction with the accompanying drawings; the data processing solution provided in the embodiment of the present application mainly includes: a method for merging / compressing the ordered data structure SSTable within the storage engine, a method for reading and writing data in the data system, etc.

[0096] See Figure 4 , Figure 4 is a flowchart of a data processing method provided by an exemplary embodiment of the present application; Figure 4 The data processing method shown mainly introduces a method for merging / compressing an ordered data structure SSTable within a storage engine. The method is applied to a data system and can be specifically executed by a storage engine (such as an LSM storage engine) loaded in a sub-data partition in the data system. The method may include but is not limited to steps S401-S402; wherein:

[0097] S401: If the storage engine loaded in the sub-data partition detects that the memory data structure is full, the data stored in the memory data structure is transferred to the first-level SSTable file.

[0098] S402: If the number of SSTable files in the first level reaches the threshold, the SSTable files in the first level are merged, and the merged SSTable files are written to the disk in the second level.

[0099] It should be noted that steps S401-S402 introduce the merging by taking the storage engine loaded in a single sub-data partition as an example; the specific implementation process of performing the merging operation on the storage engine loaded in each sub-data partition in the data system is similar. The above introduction to the process of performing the merging operation on the storage engine loaded in a single sub-data partition does not limit the embodiments of the present application.

[0100] In steps S401-S402, as described above, the embodiment of the present application divides the data partition to obtain multiple sub-data partitions belonging to the data partition. For any one of the multiple sub-data partitions, the amount of data contained in the storage engine loaded in the sub-data partition is smaller than the storage engine loaded for the entire data partition. Therefore, the amount of data that needs to be processed during merging / compression in the storage engine is smaller, thereby greatly improving the engine performance of the storage engine.

[0101] In the specific implementation, it is assumed that any one of the multiple sub-data partitions corresponding to the data partition is represented as a target sub-data partition, and the storage engine loaded in the target sub-data partition is represented as a reference storage engine (i.e., miniLSM storage engine); considering that the target sub-data partition is a sub-partition in the data partition, the data capacity of the target sub-data partition is smaller than the data capacity of the data partition. Therefore, for the target sub-data partition, the number of memory data structures MemTable in the reference storage engine (i.e., mini LSM storage engine) loaded in the target sub-data partition remains unchanged, and the data capacity of each MemTable file is reduced, and the size of each SSTable file (i.e., the amount of data that can be contained) also becomes smaller when the number of SSTable files at any level remains unchanged. In other words, after the data partition is divided, if the number of memory data structures MemTable in the first part of the reference storage engine loaded in the target sub-data partition is the same as the number of memory data structures in the first part of the storage engine loaded before the data partition is divided, and the number of ordered data structures SSTable of each level in the second part of the reference storage engine is the same as the number of ordered data structures SSTable of the corresponding level in the second part of the storage engine loaded before the data partition is split, then it can be determined that: the amount of data of the memory data structure MemTable in the reference storage engine is less than the amount of data of the memory data structure MemTable in the storage engine loaded before the data partition is split, and the amount of data of the ordered data structures SSTable of each level in the second part of the reference storage engine is less than the amount of data of the ordered data structures of the corresponding level in the storage engine loaded before the data partition is split.

[0102] The following is combined with Figure 5 The principle and corresponding beneficial effects of the above-described mini LSM including the reduction of data volume and the reduction of the size of each ordered data structure SSTable are introduced. Figure 5As shown, assuming that the data partition is not divided, the first part of the storage engine loaded inside it (that is, the standard storage engine mentioned above) contains 6 MemTable files, and the memory space occupied by the 6 MemTable files is 1200 bits; the first level in the second part contains 6 SSTable files, and the 6 SSTable files occupy 1800 bits, the second level contains 3 SSTable files, and the 3 SSTablebl e files occupy 2400 bits, and the third level contains 1 SSTable file, and the 1 SSTa ble file occupies 32 bits. If a partitioning strategy is adopted to partition the data partition into multiple sub-data partitions, and any one of the multiple sub-data partitions is represented as a target sub-data partition, then it is determined that the data capacity of the target sub-data partition is obviously smaller than the data capacity of the data partition. Therefore, for the reference storage engine loaded in the target sub-data partition, when its first part still contains 6 MemTable files, the data capacity of each MemTable file is smaller than the data capacity of a single MemTable file in the first part of the storage engine loaded when the data partition is not partitioned (e.g., the data capacity of a single MemTable file in the first part of the storage engine loaded when the data partition is not partitioned is 200 bits, then the data capacity of the MemTable file in the reference storage engine is smaller than 200 bits). Similarly, for the reference storage engine loaded in the target sub-data partition, when the number of SS Table files contained in each level in its second part remains unchanged, the amount of data contained in each SSTable file is smaller than the amount of data contained in the SSTable file of the corresponding level in the second part of the storage engine loaded when the data partition is not partitioned.

[0103] Based on the characteristics of the reference storage partition loaded in the target sub-data partition described above, it can be seen that during the merging / compression process for SSTable files, since the amount of data contained in a single MemTable file is reduced, the storage engine loaded in the target sub-data partition detects that the memory data structure MemTable file is full, and when the data stored in the memory data structure MemTable file is transferred to the first-level SSTable file, the amount of data required to be transferred is reduced compared to the amount before the partition. Similarly, since the amount of data contained in a single SSTable file is reduced, the number of SSTable files in the first level reaches the threshold. When the SSTable files in the first level are merged, the amount of data that needs to be written to the second-level SSTable file after the merger is reduced compared to the amount of data before the partition.

[0104] It can be seen from this that, on the one hand, for the reference storage engine loaded in the target sub-data partition, the amount of data that needs to be processed during each merge and compression is significantly smaller than the amount of data that needs to be processed before the partition, thereby reducing the merge consumption and greatly alleviating the write amplification problem existing in the LSM storage engine; for example, when the SSTable files of an entire layer in the reference storage engine are involved in the merge, the maximum value of the merge cost is reduced, and when there is only one SSTable file in the entire layer in the reference storage engine that participates in the merge, the minimum value of the merge cost is reduced. On the other hand, since the amount of data in the SSTable files at each level of the reference storage engine is smaller, the range of data that needs to be locked during the merge and compression will also be smaller, thereby greatly reducing lock contention. On the other hand, since reducing the size of the SSTable file can reduce the minimum granularity of each merge operation, it also reduces the possibility of passive merges (that is, some data that does not need to be merged originally, but because it is in the same SSTable file as the data that needs to be merged, this data passively participates in the merge), thereby reducing the demand for system resources for the merge operation and reducing competition for system resources.

[0105] See Figure 6 , Figure 6 is a flowchart of another data processing method provided by an exemplary embodiment of the present application; Figure 6 The data processing method shown mainly introduces a method for reading and writing data in a data system. The method can be executed by a storage engine (such as an LSM storage engine) loaded in a child data partition. The method may include but is not limited to steps S601-S604; wherein:

[0106] S601: Obtain a data operation request.

[0107] S602: Performing first-level routing processing on the data system according to the data operation indicated by the data operation request, routing from the data system to the target data partition related to the target data.

[0108] S603: Perform second-level routing processing on the target data partition, routing from the target data partition to a reference sub-data partition related to the target data.

[0109] S604: Performing data operations on target data in the reference sub-data partition through the storage engine loaded in the reference sub-data partition.

[0110] In steps S601-S604, as the architecture of the data system changes, the interaction mode between the client and the data system also changes. In a specific implementation, the data system includes an access node, which is a node (or server) that exists in the data system and plays an access or proxy role between the client and the data system; under the data system framework provided in the embodiment of the present application, the client communicates data with the data node in the data system through the access node in the data system. Furthermore, the access node in the data system communicates data with the data partition in the data node, that is, the client is routed to a data partition in the data node through the access node in the data system. Furthermore, after being routed to a specific data partition, the data partition communicates data with the sub-data partition belonging to the data partition to implement data operations (such as data reading and writing operations) in the sub-data partition.

[0111] From the above description of the interaction between the client and the data system, it can be seen that the embodiment of the present application provides a data processing method of secondary routing based on the data system; the first level routing in the secondary routing includes the routing process from the access node in the data system to the data partition in the data node, and the second level routing includes the routing process from the data partition to a sub-data partition corresponding to the data partition. For example, the schematic diagram of the secondary routing can be seen in Figure 7 ;like Figure 7 As shown, assuming that the client has a need to perform data operations in the data system, the client can send a data operation request for the target data to the data system, and the data operation request is used to request to perform data operations on the target data in the data system. Among them, data operations may include modification operations, write operations or read operations on the target data, and the target data can be any data; the embodiment of the present application does not limit the specific content of the data operation and the target data. For example, the target data is data stored in a sub-data partition in the data system, then the data operation can be a read operation or modification operation on the target data; for another example, the target data is data to be written to a sub-data partition in the data system, then the data operation can be a write operation to write the target data to the sub-data partition.

[0112] Then, after obtaining the data operation request sent by the client, the access node in the data system can perform first-level routing processing on the data system according to the data operation request for the data operation indicated by the target data. The first-level routing processing is intended to route from the data system to the target data partition related to the target data. Among them, the correlation between the target data and the target data partition mentioned above may include: when the data operation on the target data is a read operation or a modify operation, the target data is the data stored in the target data partition, so that the data operation can be performed on the target data originally stored in the target data partition in response to the data operation request indicating the data operation. Alternatively, when the data operation on the target data is a write operation, it means that the user wants to write the target data as new data to the data system, then the target data is the data to be stored in the target data partition. In detail, the specific implementation process of the first-level routing processing may include: each data node in the data system caches an attribute list, which includes one or more fields, and the fields are obtained based on the metadata of the data stored in the data node. Therefore, when performing the first-level routing processing in the data system in response to the data operation request, the first field is specifically determined from the attribute list, and the first field is used to indicate the routing path of the target data partition related to the target data from at least one data partition in the data node; in this way, routing can be performed from at least one data partition in the data node to the target data partition according to the routing path indicated by the first field.

[0113] Furthermore, after determining the target data partition related to the target data from the data node, it also supports second-level routing processing of the target data partition, which is intended to route from the target data partition to the reference sub-data partition related to the target data (such as any one of the multiple sub-data partitions, which can be the target sub-data partition mentioned above). Similar to the correlation between the target data and the target data partition described above, the correlation between the target data and the reference sub-data partition may include: when the data operation on the target data is a read operation or a modify operation, the target data is the data stored in the reference sub-data partition, so that the data operation can be performed on the target data originally stored in the reference sub-data partition in response to the data operation request indicating the data operation. Alternatively, when the data operation on the target data is a write operation, it means that the user wants to write the target data as new data to the data system, then the target data is the data to be stored in the reference sub-data partition. Among them, the target data partition contains the reference sub-data partition. In detail, the specific implementation process of the second-level routing processing may include: determining a second field from the attribute list cached in the data node, and the second field is used to determine the routing path of the reference sub-data partition related to the target data from the target data partition; and routing from multiple sub-data partitions in the target data partition to the reference sub-data partition according to the routing path indicated by the second field.

[0114] Based on the above discussion of secondary routing, it can be seen that secondary routing involves routing sub-data partitions within the same data partition. Routing sub-data partitions within the same data partition consumes relatively little resources, and the routing cost is negligible. Therefore, the secondary routing method provided by the embodiments of the present application does not increase routing overhead compared to traditional primary routing methods. Furthermore, because secondary routing involves routing between sub-data partitions within smaller data streams, it significantly improves routing speed and efficiency compared to routing within data partitions within larger data streams.

[0115] The following describes the specific processes of reading and writing target data within the framework of the data system, taking the data operations described above as read operations for target data (i.e., reading data from the data system) and write operations for target data (i.e., writing data to the data system).

[0116] (1) Write data in the data system.

[0117] In a specific implementation, when a user wants to write data to a data system, the user can send a data operation request to the data system through a client. In this case, the data operation request is used to request that a write operation on the target data be performed in the data system. Then, when the data system is a distributed data system, an access node in a cluster in the data system receives the data operation request sent by the client. In response to the data operation request and the attribute list cached by each data node in the cluster, the access node finds a data node related to the target data (i.e., the data node to which the target data is desired to be written) from at least one data node included in the cluster, and then routes from the data node to the target data partition related to the target data to complete the first-level routing. At this time, the access node can forward the target data to the target data partition.

[0118] Furthermore, the reference sub-data partition (or mini LSM) to which the target data belongs can be found from the multiple sub-data partitions in the target data partition according to the attribute list to complete the secondary routing; after finding the reference sub-data partition related to the target data, the target data can be written to the reference sub-data partition. Among them, the structure of the mini LSM loaded in the reference sub-data partition is the same as the structure of the standard storage engine (such as the LSM storage engine loaded before the data partition is divided), both of which include a first part and a second part; therefore, when writing the target data to the reference sub-data partition, specifically, the target data is first written to the memory structure data MemTable in the first part of the storage engine loaded in the reference sub-data partition, and when the memory structure data MemTable is full, the data in the memory structure data MemTable is written to the disk in the ordered data structure SSTable in the first level Level 0 of the second part.

[0119] It can be seen from this that since the reference sub-data partition is obtained by dividing it from the data partition, the data capacity of the reference sub-data partition is smaller than the data capacity of the data partition, and thus the data volume of a single SSTable file in the reference sub-data partition is smaller than the data volume of a single SSTable file in the standard LSM; this reduces the amount of data merged when the LSM storage engine in the reference sub-data partition performs merging and compression after the target data is written to the reference sub-data partition, that is, the amount of data for each merge operation is reduced, alleviating the write amplification problem; and, the data range that needs to be locked during merging and compression is smaller, thereby reducing lock contention.

[0120] (2) Read data in the data system.

[0121] In a specific implementation, when a user wants to read / query data in a data system, the user can send a data operation request to the data system through a client. In this case, the data operation request is used to request a read operation on the target data in the data system. Then, when the data system is a distributed data system, an access node in a cluster in the data system receives the data operation request sent by the client. In response to the data operation request and the attribute list cached by each data node in the cluster, the access node finds a data node related to the target data (i.e., the data node to which the target data belongs) from at least one data node included in the cluster, and then routes from the data node to the target data partition related to the target data to complete the first-level routing. At this time, the access node can forward the data operation request to the target data partition.

[0122] Furthermore, the reference sub-data partition (or mini LSM) can be found from multiple sub-data partitions in the target data partition according to the attribute list to complete the secondary routing; after finding the reference sub-data partition related to the target data, the target data can be queried in the target data partition, and the queried target data can be returned to the client to help the user query data from the data system. Among them, the structure of the mini LSM loaded in the reference sub-data partition is the same as the structure of the standard storage engine, both of which include a first part and a second part; therefore, when querying data in the reference sub-data partition, specifically, the target data is first queried in the first part of the storage engine loaded in the reference data partition. If the target data is stored in the first part, the queried target data is directly returned to the client; if the target data is not queried in the first part, the second part is queried, specifically, the filter corresponding to each SSTable file is filtered, and when it is determined that the target data is in the SSTable file, the index table of the SSTable file is queried to find the index key corresponding to the target data from the index table, and then the target data is found from the SSTable file according to the index key and returned to the client.

[0123] It can be seen that since the embodiment of the present application divides the data partition into sub-data partitions (minipartitions), the amount of data loaded in the min LSM engine in each sub-data partition is further reduced; when the number of SSTable files contained in each level of the mini LSM engine is the same as the number of SSTable files contained in the corresponding level of the standard LSM engine, the number of SSTable files required for data query remains unchanged, and data query only needs to be performed from the SSTable files with smaller data volume, thereby improving data query efficiency without causing engine read amplification problems.

[0124] In summary, based on the relevant introduction of the data system and data processing method provided in the embodiments of the present application in conjunction with the accompanying drawings, the data system and data processing method provided in the embodiments of the present application have at least the following significant effects:

[0125] ① It will not increase the complexity of cluster management. As mentioned above, the multiple sub-data partitions obtained after dividing the data partition in the embodiment of the present application still belong to the data partition, and the data migration between clusters still uses the data partition as the migration granularity; for the cluster, the number of data partitions that need to be managed has not changed. Compared with the mainstream method of increasing the number of partitions in the cluster by reducing the data capacity of a single data partition (see Figure 8 , reducing the data capacity of each partition in the cluster will lead to an increase in the number of partitions in the cluster while the data capacity of the cluster remains unchanged, or, while the number of partitions managed by the cluster remains unchanged, the data capacity that the cluster can manage will be reduced). In terms of this, the embodiments of the present application will not increase the number of partitions managed by the cluster, and thus will not increase the rules and complexity of cluster management.

[0126] ② Reduce the cost of merging and compression of the LSM storage engine. As mentioned above, since the data partition partition is further divided into smaller sub-data partitions, for the min LSM engine loaded in a single sub-data partition, the amount of data for the merge operation is reduced when performing SSTable merging and compression. For example, when all SSTable files at any level in the mini LSM are merged, the maximum merge cost is reduced, and when one SSTable file at any level in the mini LSM is merged, the minimum merge cost is reduced, thereby alleviating the write amplification problem existing in the LSM engine. In addition, since the amount of data in a single SSTable file is reduced, the range of data that needs to be locked during merging and compression is smaller, reducing lock contention. In addition, considering that some data in the SSTable file does not need to be merged, but is in the same SSTable file as the data that needs to be merged in the SSTable file, passive merging occurs; and after the embodiment of the present application reduces the amount of data in the SSTable file by dividing the data partition, the amount of data passively merged in the SSTable is also reduced, thereby reducing passive merging while reducing resource contention.

[0127] ③ There is no read amplification problem, and query efficiency is improved. As mentioned above, the embodiment of the present application divides the data partition into sub-data partitions (mini partition), further reducing the amount of data in the min LSM engine loaded in each sub-data partition. Then, when the number of SSTable files contained in each level of the mini LSM engine is the same as the number of SSTable files contained in the corresponding level of the standard LSM engine, compared with the mainstream method of increasing the number of SSTables by reducing the size of SSTables (such as Figure 9 As shown in the figure, the size of the SSTable in the LSM is reduced, but the number of SSTables is increased). In the embodiment of the present application, the number of SSTable files required to be queried when performing data query remains unchanged, and data query only needs to be performed from SSTable files with smaller data volume, thereby improving data query efficiency without causing engine read amplification problems.

[0128] Furthermore, embodiments of the present application support fine-tuning of data systems based on specific business scenarios, so that the fine-tuned data systems can meet the business requirements of specific business scenarios. The following are three exemplary fine-tuning processes for fine-tuning data systems to meet the business requirements of specific business scenarios; among them:

[0129] (1) If a specific business scenario does not focus on the read amplification problem of the LSM storage engine, then the embodiment of the present application supports further reducing the size of each ordered data structure SSTable in the architecture of the proposed data system to further alleviate the write amplification problem of the LSM storage engine and further reduce the merge cost.

[0130] In a specific implementation, the process of fine-tuning the data system to reduce the size of the ordered data structure SSTable may include: assuming that the number of ordered data structures at any level in the second part (i.e., disk) of the reference storage engine (i.e., LSM storage engine) loaded in any sub-data partition is N, where N is an integer greater than 1; then, while the data capacity of any level remains unchanged, the size of each of the N ordered data structures may be reduced, and the amount of data in the ordered data structures after the size reduction is reduced, so as to split the N ordered data structures into M ordered data results, where M is an integer greater than N. In this way, the amount of data of any one of the M ordered data structures is less than the amount of data of any one of the N ordered time results.

[0131] For example, in the data system provided in the embodiment of the present application (such as Figure 2 Based on the data system shown in the figure, the architecture diagram of the new data system obtained by further reducing the size of the ordered data structure SSTable can be seen in Figure 10 .like Figure 10 As shown, the data partition in the data node is split to obtain multiple sub-data partitions, and after the storage engine is loaded for each sub-data partition, it also supports reducing the size of the ordered data structure SSTable included in the storage engine in each sub-data partition (that is, reducing the data volume of the SSTable file); as the size of the ordered data structure SSTable decreases, the number of ordered data structures SSTable included in each level of the storage engine increases. In specific business scenarios where read amplification is not a concern, the increase in the number of ordered data structures SSTable can be ignored, but the reduction in the size of the ordered data structure SSTable can further reduce the amount of data for each merge operation during the merge and compression process, thereby further alleviating the write amplification problem and reducing the merge cost.

[0132] It should be noted that in Figure 10 Under the architecture of the data system shown in the figure, the merging and data operations performed in the sub-data partitions of the data system are similar to the above-mentioned Figure 2 The processes of merging and data operations performed by the data system shown in the figure in the sub-data partition are similar, which can be seen in the previous Figure 4 and Figure 6 The relevant description of the specific implementation process of the embodiments will not be repeated here.

[0133] (2) If the cluster size is not a concern in a specific business scenario, the embodiments of the present application support further reducing the size of the data partitions in the proposed data system architecture to further alleviate the write amplification problem of the LSM storage engine and further reduce the merging cost.

[0134] In a specific implementation, each data partition in at least one data partition in a data node can be divided, and each data partition is divided into multiple reference data partitions. The data capacity of each reference data partition is smaller than the data capacity of the data partition to which the reference data partition belongs, so as to achieve the purpose of reducing the size of the reference data partition. Furthermore, the embodiment of the present application supports using the divided reference data partition as the data partition described above, dividing each reference data partition into multiple sub-data partitions, and loading a storage engine for each sub-data partition, and multiple sub-data partitions belong to the corresponding reference data partition; in this way, the reference data partition is still used as the management granularity (such as the data migration granularity) during cluster management.

[0135] For example, in the data system provided in the embodiment of the present application (such as Figure 2 Based on the data system shown in the figure, the architecture diagram of the new data system obtained by further reducing the size of the data partition can be seen in Figure 11 .like Figure 11As shown, the data partition in the data node can be first reduced in size to obtain multiple reference data partitions corresponding to the data partition, so that the data capacity of each reference data partition is smaller than the data capacity of the data partition. Further, the reference data partition is split to obtain multiple sub-data partitions belonging to the reference data partition, and each sub-data partition is loaded with a storage engine (i.e., mini LSM). Under this implementation method, when the cluster management scale is not a concern in a specific business scenario, or when the cluster has sufficient resources for large-scale management, the size of a single ordered data structure SSTable in the sub-data partition can be further reduced by reducing the size of the data partition. Then, during the merging and compression process, the amount of data for the merge operation can be effectively reduced, the merge consumption can be reduced, and lock contention can be reduced.

[0136] It should be noted that in Figure 11 Under the architecture of the data system shown in the figure, the merging and data operations performed in the sub-data partitions of the data system are similar to the above-mentioned Figure 2 The processes of merging and data operations performed by the data system shown in the figure in the sub-data partitions are similar. For example, Figure 2 In the data system shown, when performing data operations in the data system, the data will go through two levels of routing; the first level of routing is from the access node in the data system to the data partition, and the second level of routing is from the data partition to the sub-data partition; then Figure 11 In the data system shown, a secondary route will also be passed. In this case, the first-level route in the secondary route is from the access node in the data system to the reference data partition, and the second-level route is from the reference data partition to a sub-data partition under the reference data partition.

[0137] (3) If a specific business scenario does not focus on cluster size and read amplification issues, then the embodiments of the present application support further reducing the size of data partitions and the size of the ordered data structure SSTable in the architecture of the proposed data system to further alleviate the write amplification problem of the LSM storage engine and reduce the merge cost.

[0138] In a specific implementation, the data partition in the data node can be divided, and the data partition is divided into multiple reference data partitions. The data capacity of each reference data partition is smaller than the data capacity of the data partition to which the reference data partition belongs, so as to achieve the purpose of reducing the size of the reference data partition. Further, the divided reference data partition is used as the data partition described above, and each reference data partition is divided into multiple sub-data partitions, and a storage engine is loaded for each sub-data partition, and multiple sub-data partitions belong to the corresponding reference data partition; in this way, the reference data partition is still used as the management granularity (such as data migration granularity) during cluster management. Furthermore, assuming that the number of ordered data structures at any level in the second part (i.e., disk) of the reference storage engine (i.e., LSM storage engine) loaded in the sub-data partition is N, N is an integer greater than 1; then, under the condition that the data capacity of any level remains unchanged, the size of each ordered data structure in the N ordered data structures can be reduced, and the amount of data in the ordered data structure after the size reduction is reduced, so as to achieve the result of splitting the N ordered data structures into M ordered data results, M is an integer greater than N. In this way, the data volume of any ordered data result in the M ordered data structures is smaller than the data volume of any ordered data structure of the N ordered time results.

[0139] For example, in the data system provided in the embodiment of the present application (such as Figure 2 The architecture diagram of the new data system obtained by further reducing the size of the data partition partition and the size of the ordered data structure SSTable can be seen in Figure 12 .like Figure 12 As shown, the data partition in the data node is first reduced in size to obtain multiple reference data partitions corresponding to the data partition, and then the reference data partition is split to obtain multiple sub-data partitions belonging to the reference data partition, and each sub-data partition is loaded with a storage engine (i.e., mini LSM). Furthermore, the size of the ordered data structure SSTable included in the storage engine is reduced in each sub-data partition (i.e., the amount of data in the SSTable file is reduced). In the above process, by reducing the size of the data partition and actively reducing the size of a single ordered data structure SSTable, the data flow of a single ordered data structure SSTable in the sub-data partition can be further reduced, and the amount of data for each merge operation can be further reduced during the merging and compression process, thereby further alleviating the write amplification problem, reducing the merging cost, and reducing lock contention.

[0140] It should be noted that in Figure 12 Under the architecture of the data system shown in the figure, the merging and data operations performed in the sub-data partitions of the data system are similar to the above-mentioned Figure 2The data system shown, Figure 10 The data system shown and Figure 11 In the data system shown, the processes of merging and data operations performed in the sub-data partitions are similar. Please refer to the relevant description of the specific implementation process of the aforementioned related embodiments, and no further details will be given here.

[0141] The method of the embodiment of the present application is described in detail above. In order to facilitate the above-mentioned scheme of the embodiment of the present application to be better implemented, accordingly, the device of the embodiment of the present application is provided below. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuit or memory) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the module or unit function.

[0142] Figure 13 A schematic diagram of the structure of a data processing device provided by an exemplary embodiment of the present application is shown; the data processing device can be installed in a data system for executing Figure 4 and Figure 6 Some or all of the steps in the method embodiment shown. Figure 13 , the device includes the following units:

[0143] An acquiring unit 1301 is configured to acquire a data operation request, where the data operation request is used to request execution of a data operation on target data in a data system;

[0144] The processing unit 1302 is configured to perform a first level routing process on the data system according to the data operation indicated by the data operation request, and route the data from the data system to a target data partition associated with the target data;

[0145] The processing unit 1302 is further configured to perform a second level routing process on the target data partition, routing from the target data partition to a reference sub-data partition associated with the target data;

[0146] The processing unit 1302 is further configured to perform data operations on the target data in the reference sub-data partition by using the storage engine loaded in the reference sub-data partition.

[0147] In one implementation, the data node caches an attribute list, the attribute list including one or more fields, the fields being obtained based on metadata of the data stored in the data node;

[0148] The processing unit 1302 is configured to perform first-level routing processing on the data system according to the data operation indicated by the data operation request, and specifically to:

[0149] Determine a first field from the attribute list; the first field is used to indicate a routing path of a target data partition associated with the target data from at least one data partition in the data node;

[0150] According to the routing path indicated by the first field, routing is performed from at least one data partition in the data node to the target data partition.

[0151] In one implementation, the processing unit 1302 is configured to perform the second-level routing processing on the target data partition, and when routing from the target data partition to the reference sub-data partition related to the target data, specifically to:

[0152] determining a second field from the attribute list, the second field being used to indicate a routing path for determining a reference sub-data partition related to the target data from the target data partition;

[0153] According to the routing path indicated by the second field, routing is performed from the multiple sub-data partitions in the target data partition to the reference sub-data partition.

[0154] According to one embodiment of the present application, Figure 13 The various units in the data processing device shown can be individually or entirely combined into one or several other units to form a whole, or one (or more) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the data processing device may also include other units. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the following can be executed by running on a general-purpose computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM). Figure 4 or Figure 6 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 13The data processing device shown in and the data processing method of the embodiment of the present application are implemented. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0155] In an embodiment of the present application, when the data partition of the data node in the data system is divided into sub-data partitions, when any data node receives a data operation request related to the target data sent from the outside, it can perform a first-level routing processing on the data system in response to the data operation request, aiming to find the target data partition from the data system, and the target data partition has an associated relationship with the target data (such as in a write data scenario, it is hoped that the target data will be written to the target data partition, and in a read data scenario, the target data belongs to the target data partition, etc.). Then, a second-level routing processing is performed on the target data partition to find a reference sub-data partition from the target data partition, and the reference sub-data partition has an associated relationship with the target data (similar to the above explanation of the associated relationship); since the second-level routing processing is performed in a single data partition, the routing cost is very low and will not cause problems such as waste of resources. In this way, data operations can be performed on the target data in the reference sub-data partition through the storage nodes loaded in the reference sub-data partition. It can be seen from this that since the corresponding sub-data partition contains less data, the multi-layer ordered data structure contained in the storage engine, while the number of ordered data structures in each layer remains unchanged, the data contained in a single ordered data structure is also reduced. For the storage engine, performing data operations in the corresponding sub-data partition is compared to performing data operations in the complete data partition. This effectively reduces the amount of data that the storage engine needs to process when performing data operations, thereby improving the working performance of the storage engine while ensuring the correctness of data operations.

[0156] Figure 14 FIG2 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Figure 14, the computer device includes a processor 1401, a communication interface 1402 and a computer-readable storage medium 1403. The processor 1401, the communication interface 1402 and the computer-readable storage medium 1403 can be connected via a bus or other means. The communication interface 1402 is used to receive and send data. The computer-readable storage medium 1403 can be stored in the memory of the computer device. The computer-readable storage medium 1403 is used to store computer programs. The computer programs include program instructions. The processor 1401 is used to execute the program instructions stored in the computer-readable storage medium 1403. The processor 1401 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.

[0157] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the processor 1401 are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0158] In one embodiment, the computer-readable storage medium stores one or more instructions; processor 1401 loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned data processing method embodiment. In a specific implementation, the computer device can be a data node in a data system; processor 1401 loads the one or more instructions in the computer-readable storage medium and executes the following steps:

[0159] Obtaining a data operation request, where the data operation request is used to request execution of a data operation on target data in a data system;

[0160] Performing first-level routing processing on the data system according to the data operation indicated by the data operation request, routing from the data system to a target data partition associated with the target data;

[0161] Performing a second level routing process on the target data partition, routing from the target data partition to a reference sub-data partition associated with the target data;

[0162] Data operations are performed on target data in the reference sub-data partition through the storage engine loaded in the reference sub-data partition.

[0163] In one implementation, the data node caches an attribute list, the attribute list including one or more fields, the fields being obtained based on metadata of the data stored in the data node;

[0164] The one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and, when performing the first-level routing processing on the data system according to the data operation indicated by the data operation request and routing from the data system to the target data partition associated with the target data, specifically perform the following steps:

[0165] Determine a first field from the attribute list; the first field is used to indicate a routing path of a target data partition associated with the target data from at least one data partition in the data node;

[0166] According to the routing path indicated by the first field, routing is performed from at least one data partition in the data node to the target data partition.

[0167] In one implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1401 and, when performing the second-level routing process on the target data partition and routing from the target data partition to the reference sub-data partition associated with the target data, specifically perform the following steps:

[0168] determining a second field from the attribute list, the second field being used to indicate a routing path for determining a reference sub-data partition related to the target data from the target data partition;

[0169] According to the routing path indicated by the second field, routing is performed from the multiple sub-data partitions in the target data partition to the reference sub-data partition.

[0170] In an embodiment of the present application, when the data partition of the data node in the data system is divided into sub-data partitions, when any data node receives a data operation request related to the target data sent from the outside, it can perform a first-level routing processing on the data system in response to the data operation request, aiming to find the target data partition from the data system, and the target data partition has an associated relationship with the target data (such as in a write data scenario, it is hoped that the target data will be written to the target data partition, and in a read data scenario, the target data belongs to the target data partition, etc.). Then, a second-level routing processing is performed on the target data partition to find a reference sub-data partition from the target data partition, and the reference sub-data partition has an associated relationship with the target data (similar to the above explanation of the associated relationship); since the second-level routing processing is performed in a single data partition, the routing cost is very low and will not cause problems such as waste of resources. In this way, data operations can be performed on the target data in the reference sub-data partition through the storage nodes loaded in the reference sub-data partition. It can be seen from this that since the corresponding sub-data partition contains less data, the multi-layer ordered data structure contained in the storage engine, while the number of ordered data structures in each layer remains unchanged, the data contained in a single ordered data structure is also reduced. For the storage engine, performing data operations in the corresponding sub-data partition is compared to performing data operations in the complete data partition. This effectively reduces the amount of data that the storage engine needs to process when performing data operations, thereby improving the working performance of the storage engine while ensuring the correctness of data operations.

[0171] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-mentioned data processing method.

[0172] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that includes one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs (Digital Versatile Discs)), or semiconductor media (eg, solid-state drives (SSDs)).

[0174] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technical object of a person skilled in the art that can be easily conceived of within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data system, characterized in that: The data system includes a data node, and the data node includes at least one data partition; wherein, Any one of the at least one data partitions is divided into a plurality of sub-data partitions; a data capacity of each of the plurality of sub-data partitions in the any one data partition is smaller than a data capacity of the any one data partition; the data partition is a data migration unit of the data system; Each of the multiple sub-data partitions in any one data partition is loaded with a storage engine; the storage engine loaded in any one sub-data partition is used to perform data operations on the any one sub-data partition.

2. The system according to claim 1, wherein The data system is a distributed data system, which includes multiple clusters, each of which includes the data nodes; wherein, The plurality of clusters include a first cluster and a second cluster; when data migration is required between the first cluster and the second cluster, data migration is performed using data partitions as migration units; or, The cluster includes a first data node and a second data node. When data migration is required between the first data node and the second data node, data migration is performed using data partitions as data migration units.

3. The system according to claim 1, wherein: The data system includes an access node; the client communicates data with the data node in the data system through the access node; The access node communicates data with the data partition in the data node; The data partition communicates data with the sub-data partitions within the data partition.

4. The system according to claim 1, wherein: The data capacity of any one of the data partitions is equal to the sum of the data capacities of the sub-data partitions in the plurality of sub-data partitions in the any one of the data partitions; The partitioning strategy adopted when the data partition is divided into multiple sub-data partitions includes at least one of the following: Classify according to the business attributes of the business to which the data belongs; or Divide by data attributes; or, Divide by the hash value of the data.

5. The system according to claim 1, wherein: Any one of the sub-data partitions is represented as a target sub-data partition, and the storage engine loaded in the target sub-data partition is represented as a reference storage engine; The reference storage engine includes: a first part and a second part; the first part includes multiple memory data structures, the second part is a multi-level data structure, and each level in the second part includes one or more ordered data structures; Among them, if the number of memory data structures in the reference storage engine is the same as the number of memory data structures in the storage engine loaded before the data partition is split, and the number of ordered data structures of each level in the second part of the reference storage engine is the same as the number of ordered data structures of the corresponding level in the storage engine loaded before the data partition is split, then the data volume of the memory data structure in the reference storage engine is smaller than the data volume of the memory data structure in the storage engine loaded before the data partition is split, and the data volume of the ordered data structures of each level in the second part of the reference storage engine is smaller than the data volume of the ordered data structures of the corresponding level in the storage engine loaded before the data partition is split.

6. The system according to claim 5, wherein: The number of ordered data structures at any level in the second portion of the reference storage engine is N, where N is an integer greater than 1; The N ordered data structures are split into M ordered data structures; M is an integer greater than N; The data volume of any one of the M ordered data structures is smaller than the data volume of any one of the N ordered data structures.

7. The system according to claim 5 or 6, characterized in that Each data partition in at least one of the data partitions is divided into a plurality of reference data partitions; the data capacity of the reference data partition is smaller than the data capacity of the data partition to which the reference data partition belongs; Each of the reference data partitions is divided into a plurality of sub-data partitions.

8. A data processing method, characterized in that: The method is applied to the data system according to any one of claims 1 to 7; the method comprises: Obtaining a data operation request, wherein the data operation request is used to request to perform a data operation on target data in the data system; performing first-level routing processing on the data system according to the data operation indicated by the data operation request, routing from the data system to a target data partition associated with the target data; Performing a second level routing process on the target data partition, routing from the target data partition to a reference sub-data partition associated with the target data; The data operation is performed on the target data in the reference sub-data partition by using a storage engine loaded in the reference sub-data partition.

9. The method according to claim 8, wherein The data node caches an attribute list, wherein the attribute list includes one or more fields, and the fields are obtained based on metadata of the data stored in the data node; The performing first-level routing processing on the data system according to the data operation indicated by the data operation request, and routing from the data system to a target data partition associated with the target data, includes: Determine a first field from the attribute list; the first field is used to indicate a routing path of a target data partition related to the target data determined from at least one data partition in the data node; Routing from at least one data partition in the data node to a target data partition according to the routing path indicated by the first field.

10. The method according to claim 9, wherein The performing second-level routing processing on the target data partition, routing from the target data partition to a reference sub-data partition related to the target data, includes: determining a second field from the attribute list, where the second field is used to indicate a routing path for determining a reference sub-data partition related to the target data from the target data partition; According to the routing path indicated by the second field, routing is performed from the multiple sub-data partitions in the target data partition to the reference sub-data partition.

11. A data processing device, characterized in that: The data processing device is mounted in a data system and includes: an acquiring unit, configured to acquire a data operation request, wherein the data operation request is used to request execution of a data operation on target data in the data system; the data partition being a data migration unit of the data system; a processing unit, configured to perform a first level routing process on the data system according to the data operation indicated by the data operation request, and route the data from the data system to a target data partition associated with the target data; The processing unit is further configured to perform a second level routing process on the target data partition, routing from the target data partition to a reference sub-data partition associated with the target data; The processing unit is further configured to perform the data operation on the reference data in the target sub-data partition through the storage engine loaded in the reference sub-data partition.

12. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the data processing method according to any one of claims 8 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the data processing method according to any one of claims 8 to 10.

14. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the data processing method according to any one of claims 8 to 10 is implemented.

Images