Data processing method and apparatus based on database, and cluster

Abstract

Provided in the present application are a data processing method and apparatus based on a database, and a cluster. The method comprises: receiving an access request, wherein the access request is used for requesting access to target data in a data table, which target data has a target affinity identifier; when a plurality of affinity identifiers comprise the target affinity identifier, identifying from among a plurality of partitions a first partition corresponding to the target affinity identifier; and on the basis of the access request, accessing the target data in the first partition. The method can improve the access performance of a data table.

Description

Database-based data processing method and device and cluster

[0001] The present application claims priority to the Chinese patent application No. 202410468073.7, filed on April 16, 2024, entitled "Database-based data processing method, device and cluster", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of database, and in particular to a database-based data processing method, device and cluster. BACKGROUND

[0003] In cloud services, a plurality of tenants share a data platform (referred to as data multi-tenancy) technology is a key technology to realize reasonable reuse of resources. In the data multi-tenancy technology, in order to improve resource utilization, database resources are reused among multiple tenants. Specifically, the data of multiple tenants is stored in the same data table of the database.

[0004] When a tenant accesses the data in the data table, the data table is locked. Since the data table stores the data of multiple tenants, locking the data table affects the access of other tenants to the data in the data table, resulting in a decline in the access performance of the data table. SUMMARY

[0005] The present application provides a database-based data processing method, device and cluster, which can improve the access performance of the data table.

[0006] In a first aspect, a database-based data processing method is provided, the database including a data table, the data table including a plurality of partitions, the plurality of partitions and a plurality of ownership identifiers corresponding one-to-one, a partition being configured to store data having an ownership identifier corresponding to the partition; the method comprising: receiving an access request, the access request being configured to request access to target data in the data table, the target data having a target ownership identifier; when the plurality of ownership identifiers includes the target ownership identifier, identifying a first partition corresponding to the target ownership identifier among the plurality of partitions; and accessing the target data in the first partition based on the access request. Wherein the different partitions are physically isolated. In one example, the ownership identifier can be a tenant identifier, i.e., different ownership identifiers in the plurality of ownership identifiers are different tenant identifiers. In another example, the ownership identifier can be a data type, i.e., different ownership identifiers in the plurality of ownership identifiers correspond to different data types.

[0007] In the method, data of different home identities is stored in different partitions in the data table, when data with a certain home identity is accessed, the data is accessed in the range of the partition corresponding to the home identity, instead of being accessed in the whole range of the data table, so that the access to the data does not affect the data access of other partitions, the concurrency of the data access of the data table is improved, and the access performance of the data table is improved.

[0008] In a possible implementation, the first partition has an index, the index is created based on data in the first partition; and the target data is accessed in the first partition based on the access request, including: querying the target data in the first partition based on the index; and the target data is accessed based on the access request.

[0009] The partition is created with an index, the target data can be queried based on the index of the partition, and the query efficiency of the target data is improved. The partition is created based on data in the partition, and is irrelevant to other partitions, so that data insertion or deletion in the index only needs to lock the data in the partition, and does not affect other partitions.

[0010] In a possible implementation, the plurality of partitions and the plurality of storage spaces are in one-to-one correspondence, and the storage space corresponding to the partition is used to store data with a home identity corresponding to the partition; the first partition corresponding to the target home identity is identified from the plurality of partitions, including: identifying the storage space corresponding to the first partition from the plurality of storage spaces; and the target data is accessed in the first partition based on the access request, including: accessing the target data in the storage space corresponding to the first partition based on the access request.

[0011] The data in different partitions is in different storage spaces, and physical isolation between the data in different partitions is implemented. The access to the data in the first partition is performed in the storage space corresponding to the first partition, and does not affect other partitions, so that the access concurrency of the data packet in which the first partition is located can be improved, and the access performance of the data packet is improved.

[0012] In a possible implementation, the plurality of partitions and the plurality of files are in one-to-one correspondence, and the file corresponding to the partition is used to store data with a home identity corresponding to the partition; the first partition corresponding to the target home identity is identified from the plurality of partitions, including: identifying the file corresponding to the first partition from the plurality of files; and the target data is accessed in the first partition based on the access request, including: accessing the target data in the file corresponding to the first partition based on the access request.

[0013] The data in different partitions is in different files, and physical isolation between the data in different partitions is implemented. The access to the data in the first partition is performed in the file corresponding to the first partition, and does not affect other partitions, so that the access concurrency of the data packet in which the first partition is located can be improved, and the access performance of the data packet is improved.

[0014] In a possible implementation, the method further includes: when the plurality of home identities does not include the target home identity, creating a second partition corresponding to the target home identity in the data table; and based on the access request, accessing the target data in the second partition.

[0015] When new data of a home identity needs to be stored, a partition is created in the data table, and the data is stored in the created partition. In this way, there is no impact on the existing partitions in the data table.

[0016] In a second aspect, a database-based data processing apparatus is provided, the database including a data table, the data table including a plurality of partitions, the plurality of partitions and a plurality of home identities corresponding to each other, and the partitions being configured to store data having a home identity corresponding to the partition; the apparatus including: a receiving module configured to receive an access request, the access request being configured to request access to target data in the data table, the target data having a target home identity; an identifying module configured to, when the plurality of home identities includes the target home identity, identify a first partition corresponding to the target home identity from the plurality of partitions; and an accessing module configured to, based on the access request, access the target data in the first partition.

[0017] In a possible implementation, the first partition has an index, the index being created based on data in the first partition; and the accessing module is configured to: based on the index, query the target data in the first partition; and based on the access request, access the target data.

[0018] In a possible implementation, the plurality of partitions and a plurality of storage spaces correspond to each other, a storage space corresponding to the partition being configured to store data having a home identity corresponding to the partition; the identifying module is configured to: identify a storage space corresponding to the first partition from the plurality of storage spaces; and the accessing module is configured to: based on the access request, access the target data in the storage space corresponding to the first partition.

[0019] In a possible implementation, the plurality of partitions and a plurality of files correspond to each other, a file corresponding to the partition being configured to store data having a home identity corresponding to the partition; the identifying module is configured to: identify a file corresponding to the first partition from the plurality of files; and the accessing module is configured to: based on the access request, access the target data in the file corresponding to the first partition.

[0020] In a possible implementation, the apparatus further includes a creating module; the creating module is configured to: when the plurality of home identities does not include the target home identity, create a second partition corresponding to the target home identity in the data table; and the accessing module is further configured to: based on the access request, access the target data in the second partition.

[0021] In a possible implementation, the different home identities in the plurality of home identities are different tenant identities, or the different home identities in the plurality of home identities correspond to different data types.

[0022] In a third aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method provided in the first aspect.

[0023] In a fourth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0024] In a fifth aspect, a computer program product including instructions is provided, when the instructions are executed by a computing device cluster, the computing device cluster executes the method provided in the first aspect.

[0025] The beneficial effects of the second aspect to the fifth aspect can be referred to the description of the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0026] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0027] FIG. 2 is a schematic diagram of a data table provided by an embodiment of the present application;

[0028] FIG. 3 is a schematic diagram of a data table provided by an embodiment of the present application;

[0029] FIG. 4 is a schematic diagram of index creation provided by an embodiment of the present application;

[0030] FIG. 5 is a schematic diagram of index creation provided by an embodiment of the present application;

[0031] FIG. 6 is a flowchart of a data processing method provided by an embodiment of the present application;

[0032] FIG. 7 is a schematic diagram of a data processing method provided by an embodiment of the present application;

[0033] FIG. 8 is a schematic diagram of a data processing method provided by an embodiment of the present application;

[0034] FIG. 9 is a schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0035] FIG. 10 is a schematic diagram of a computing device provided by an embodiment of the present application;

[0036] FIG. 11 is a structural diagram of a computing device cluster according to an embodiment of the present application;

[0037] FIG. 12 is a structural diagram of a computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The schemes provided by the embodiments of the present application will be described below with reference to the drawings. In the embodiments of the present application, "multiple" refers to two or more, and "various" refers to two or more. "First", "second", and the like are only used to distinguish similar objects, and do not necessarily describe a specific order or number of objects.

[0039] To facilitate understanding of the schemes provided by the embodiments of the present application, the technical terms that may be involved in the embodiments of the present application are first introduced.

[0040] Data table: a table participating in data storage or management in a database. The data table has a table header and the like. The database manages metadata and templates in the granularity of data tables. That is, the same data table corresponds to the same metadata and the same template, and different data tables correspond to different metadata and different templates.

[0041] Logical data table: also referred to as a logical table, is a kind of data table in a database. The logical data table is a virtual table and does not store data.

[0042] Physical data table: also referred to as a physical table, is a table that can store data in a database. Data can be recorded in the physical data table to store the data into the physical data table.

[0043] Template: refers to a statement template, including an insertion statement template, a query statement template, and the like. Taking the query statement template as an example, when querying target data, the constant in the query statement template is modified into a variable related to the target data, and a query statement for querying the target data is obtained.

[0044] Metadata of a data table: refers to data describing the data table. The metadata of the data table can include description information of a table header of the data table and the like.

[0045] Ownership identifier: is an identifier used to indicate which tenant or which type of data the data belongs to. The ownership identifier can be a tenant identifier, used to indicate that the data belongs to the tenant corresponding to the tenant identifier. The ownership identifier can be an identifier of a data type, used to indicate that the data is data of the data type. For example, the data type can include write-intensive and read-intensive.

[0046] Flexible field: refers to a table header with a non-fixed meaning in a data table. When a table header of a data table is a flexible field, the meaning of the table header under different tenants can be different.

[0047] The three common modes of cloud services are infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Among them, the infrastructure and application management of SaaS are provided by the cloud service provider, which enables the cloud service provider to reuse resources and solutions among tenants, thereby greatly reducing the operating costs of the provider and the costs of tenants renting cloud services.

[0048] Multi-tenant shared data platform technology, referred to as data multi-tenancy technology, can achieve reasonable reuse of resources, thereby achieving the effect that the more tenants, the lower the cost. For a database management system (DMBS), reasonable reuse of resources includes data table reuse, that is, storing data of multiple tenants in the same data table. The multi-tenant reuse of the data table is implemented through row-level data isolation.

[0049] In order to implement data table reuse, a unique tenant identifier (renter ID) is allocated to each tenant, and the table and field of each tenant are virtual. Whenever a new tenant is added, the database management system does not create a new data table for the tenant in the database (DB), but inserts the data of the tenant and the tenant identifier of the tenant into the existing data table. When the data of a tenant needs to be queried, the tenant identifier of the tenant is used as a filter condition to query the data of the tenant in the data table. Moreover, when a tenant adds a field, the database management system does not add a field in the data table, but establishes a mapping relationship between the field to be added by the tenant and the flexible field of the data table, and the column corresponding to the flexible field to which the field to be added by the tenant is mapped is used to record the data of the field to be added by the tenant. The mapping relationship between the field to be added by the tenant and the flexible field of the data table can be recorded in a metadata engine.

[0050] In related technologies, in order to avoid conflicts caused by multiple tenants accessing data in a data table at the same time, when a tenant accesses data in a data table, the data table is locked, and the lock only allows the current tenant to access the data table and prohibits other tenants from accessing the data table. Therefore, multiple tenants can only access the data table in series, resulting in low access efficiency and poor performance of the data table.

[0051] In particular, to ensure the efficiency of tenant access to data, tenant-level indexes of data tables are created. Different tenants in the same data table have different requirements for the number of indexes, flexible field types, etc. The first flexible field of tenant 1 is of timestamp type, requiring range condition queries. The first flexible field of tenant 2 is of text type, requiring the creation of an inverted index suitable for full-text search. The first flexible field of tenant 3 is just a descriptive field and does not require an index. Therefore, for a multi-tenant shared data table, when creating an index, tenant data needs to be isolated, that is, a tenant-level index only takes effect on data of a specific value of the tenant.

[0052] In one scheme, tenant-level predicate partial indexes are created for data tables. Specifically, an equality filter condition on a tenant identification field is specified when creating an index, so that data with the same tenant identification is inserted into the same index. In this scheme, when tenant 1 data needs to be inserted into a data table, all indexes of all tenants of the data table need to be traversed, and it is determined in turn whether the traversed index belongs to tenant 1. The process of traversing all tenant indexes requires locking all tenant indexes and reading the metadata of the indexes. In addition, when tenant 1 data needs to be queried in a data table, all tenant indexes also need to be traversed, and the tenant identification of tenant 1 is compared with the predicates of all partial predicate indexes. When a predicate that identifies the same tenant identification as tenant 1 is obtained, the corresponding index path is generated. This process also requires locking all tenant indexes and reading the metadata of the indexes.

[0053] Locking all tenant indexes causes all tenant indexes to be inaccessible to tenants other than tenant 1, which reduces data access performance. Moreover, as the number of tenants in the data table increases, data access performance will continue to decline. Specifically, locking all tenant indexes requires a large number of lock resources. Traversing all tenant indexes and reading metadata requires caching all tenant indexes and metadata, which is likely to cause the index cache and metadata cache to be hit, resulting in a sharp decline in performance.

[0054] In addition, this scheme cannot take advantage of the template general plan of database precompiled statements to optimize query performance. Specifically, a statement precompilation interface is usually provided by a database to improve the execution performance of high-frequency template statements. If the access request of a client hits a compiled statement template, the cached general execution plan can be directly executed, thereby realizing lightweight structural query language (SQL) parsing. However, for a predicate partial index, the general plan cannot determine in the plan generation stage whether the generated index plan is suitable for the actual called tenant identification.

[0055] In addition, the tenant data migration capability is poor in the scheme. Since the tenant data uses function row-level isolation, the tenant data is cross-distributed in different data files. When the tenant data is cleaned and migrated, multiple data files need to be scanned and modified row by row, and the overhead is high.

[0056] In another scheme, after the index field of the tenant in the data table is converted into the target data type, the indexes of the field with the same target data type are copied into the same perspective table, and the indexes of the field with different target data types are copied into different perspective tables. When the data in the data table is accessed, such as added, deleted, or modified, the business arrangement layer expands a single data access operation statement into multiple maintenance statements of the data table and the perspective table, and places the multiple maintenance statements in the same transaction block to ensure the atomicity between the data table and the perspective table. When the data in the data table is queried, the business arrangement layer parses the query task in the data table into a query statement of the perspective table according to the semantic of the tenant query, and uses the index to query the data.

[0057] The scheme needs the business arrangement layer to expand the statement and parse the task, and the complexity is high. In addition, the indexes of the field with different target data types are copied into different perspective tables, and the target data type of the field of the tenant's index is used to identify the perspective table corresponding to the tenant. This operation is tedious, and different perspective tables need different statement expansion functions and task parsing functions, that is, the business arrangement layer needs to adapt to different perspective tables, resulting in high expansion cost and parsing cost. In addition, since the data table may need multiple perspective tables, the number of system objects and templates of the database doubles, which increases the cache pressure and causes the concurrent performance to decline.

[0058] Embodiments of the present application provide a database-based data processing method. The method can set a data table in a database, and divide the data table into multiple partitions, which are physically isolated between different partitions. Each partition is used to store data under the same ownership identifier. When accessing target data, the partition where the target data is located can be obtained based on the ownership identifier of the target data, and then the target data is accessed in the partition. In this way, data access is performed based on the partition, and when data access is performed in a certain partition, other partitions are not affected by the data access, and data access can be performed in parallel between different partitions, thereby improving the concurrency of data access of the data table and improving the access performance of the data table.

[0059] Next, the method provided by the embodiments of the present application will be described in detail.

[0060] FIG. 1 shows a system architecture that can be used to implement the method. The system architecture includes a server 100 and a client 200. The server 100 is deployed with a database 300. The database 300 can include at least one data table, such as a data table 310, a data table 320, etc. The database 300 further includes a database management system 330. The database management system 330 can manage the data tables in the database 300. The client 200 can issue an access request, and the database management system 330 can receive the access request and perform relevant data access based on the access request.

[0061] The data tables in the database 300 can be divided into multiple partitions. Referring to FIG. 2, taking the data table 310 as an example, the data table 310 can be divided into a partition 311, a partition 312, etc. Each partition corresponds to a home identity, and the partition is used to store data having the home identity corresponding to the partition. Different partitions of the same data table are physically isolated. For example, different partitions are stored in different storage spaces. For example, different partitions are stored in different files. In this way, access to one partition will not affect another partition, and data access can be performed in parallel between different partitions, thereby improving the access performance of the data table.

[0062] The data table 310 is presented to the tenant. Therefore, the data actually accessed by the tenant is in the partition 311 or the partition 312, and the tenant can initiate an access request for the data table 310 without the tenant identifying the partition where the target data of the access request is located. Therefore, the user experience is improved.

[0063] In addition, the partition 311, the partition 312, etc. belong to the data table 310, and therefore, the partition 311, the partition 312, etc. share the metadata and templates of the data table 310, thereby eliminating the need for separate metadata and template management for each partition, saving the management overhead of the database.

[0064] Referring to FIG. 3, the data table 310 is partitioned according to rows. In some embodiments, the server 200 is deployed with an objective query language (OQL) engine. The OQL engine can issue a multi-tenancy system initialization statement, which instructs the database management system 330 to create the data table 310, the data table 320, etc. in the database 300, and instructs the database management system 330 to partition the data table 310, the data table 320, etc. according to the home identity. The data table 310 can be one of a tenant entity table and a tenant extension table, and the data table 320 can be the other of the tenant entity table and the tenant extension table. The tenant entity table refers to a data table with fixed table header meaning, i.e., the meaning of the table header is the same for different tenants. The tenant extension table refers to a data table with flexible table header, i.e., the meaning of the table header can be different for different tenants.

[0065] In one example, the data table 310 can be partitioned in a list partition manner. For example, even rows in the data table 310 are mapped to the home identity A1, and odd rows are mapped to the home identity A2, then the even rows of the data table 310 form a partition corresponding to the home identity A1, and the odd rows of the data table 310 form a partition corresponding to the home identity A1. In one example, the data table 310 can be partitioned in a range partition manner, for example, rows 1-100 in the data table 310 are divided into the partition 311, and rows 101-200 in the data table 310 are divided into the partition 312.

[0066] In some embodiments, the data table 310 can be a logical table, and the partition 311 and the partition 312 can be physical tables mapped to the data table 310.

[0067] In some embodiments, the data table 310 can be referred to as a set of physical tables, and the physical tables in the set share metadata and templates, etc. The partition 311 and the partition 312 are physical tables in the set.

[0068] In some embodiments, an index can be created based on data in a partition. Since different partitions are independent of each other, when an index is created based on data in a partition, only the current partition needs to be locked, and other partitions do not need to be locked. In this way, other partitions are not affected, and the business of other partitions is not blocked.

[0069] In some embodiments, the data table 310 can include a flexible field. When an index is created for the flexible field, the partition in the data table 310 converts the expression index according to the data type of the flexible field in the current partition, without considering the data type of the flexible field in other partitions. The data type corresponding to the flexible field or the table header is the data in the column where the flexible field or the table header is located.

[0070] In one example, the table header V1, the table header V2, and the table header V4 of the data table 310 can be set as flexible fields. The data type corresponding to the table header V1 in the partition 311 can be set as int8, the data type of the data corresponding to the table header V2 can be set as timestamp, and the data type corresponding to the table header V4 can be set as numeric. For the partition 311, the index 3111 and the index 3112 shown in FIG. 4 can be generated.

[0071] In one example, the table header V1 and the table header V2 of the data table 310 can be set as flexible fields. The data type corresponding to the table header V1 in the partition 312 can be set as timestamp, and the data type of the data corresponding to the table header V2 can be set as varchar. For the partition 312, the index 3121 shown in FIG. 5 can be generated.

[0072] In some embodiments, the ownership identifier can be a data type. Generally, the data type herein includes read-intensive and write-intensive. When the ratio of the occurrence frequency of the read operation on the data to the occurrence frequency of the write operation on the data is greater than a preset threshold, the data type of the data is read-intensive. When the ratio of the occurrence frequency of the write operation on the data to the occurrence frequency of the read operation on the data is greater than a preset threshold, the data type of the data is write-intensive.

[0073] An index corresponding to the ownership identifier of the partition can be created. For example, when the ownership identifier of the partition is read-intensive, a read-friendly index is created for the partition. The read-friendly index refers to an index suitable for read operation, so that the read size, read latency, and read operation experience are good. Common read-friendly indexes include indexes in a log structured merge tree (LSM tree) structure. For another example, when the ownership identifier of the partition is write-intensive, a write-friendly index is created for the partition. The write-friendly index refers to an index suitable for write operation, so that the write size, write latency, and write operation experience are good. Common write-friendly indexes include indexes in a B+ tree structure.

[0074] The above examples introduce the system architecture provided by the embodiments of the present application. Next, the data processing method provided by the embodiments of the present application is described in combination with the system architecture.

[0075] The method can be performed by the server 100. For example, the method can be performed by the database management system 330 in the server 100. As described above, the database 300 deployed in the server 100 includes the data table 310, where the data table 310 includes a plurality of partitions 311 and 312, and the plurality of partitions and a plurality of home identifiers are in one-to-one correspondence, and each partition is configured to store data having a home identifier corresponding to the partition. As shown in FIG. 6, the method includes the following steps.

[0076] In step 601, an access request is received, where the access request is configured to request to access target data in the data table 310, and the target data has a home identifier A1.

[0077] The client 200 can send the access request in response to an operation of a tenant. The access request includes related information of data to be accessed by the access request, and the related information includes the home identifier A1, which is a home identifier of the data to be accessed by the access request. The data to be accessed by the access request can be referred to as target data.

[0078] As described above, the server 100 presents the data table 310 to the client 200, so that the client 200 can send the access request to the data table 310, that is, send an access request to access data in the data table 310.

[0079] In some embodiments, the access request can be an insertion request or a write request, which is configured to insert the target data. In some embodiments, the access request can be a read request, which is configured to read the target data. In some embodiments, the access request can be a modification request, which is configured to modify the target data. In some embodiments, the access request can be a deletion request, which is configured to delete the target data.

[0080] In some embodiments, the home identifier can be a tenant identifier of a tenant, that is, the data having the home identifier is data of a tenant corresponding to the home identifier.

[0081] In some embodiments, the home identifier can be a data type. Generally, the data type here includes read-intensive and write-intensive.

[0082] In step 602, it is determined whether the plurality of home identifiers includes the home identifier A1.

[0083] The plurality of home identifiers are home identifiers corresponding to the partitions, and the home identifier A1 is in the plurality of home identifiers, that is, the plurality of home identifiers includes the home identifier A1, which indicates that the data table 310 includes a partition corresponding to the home identifier A1. When the home identifier A1 is not in the plurality of home identifiers, that is, the plurality of home identifiers does not include the home identifier A1, which indicates that the data table 310 does not include a partition corresponding to the home identifier A1.

[0084] In some embodiments, when the data table 310 is partitioned based on the home identity, the home identity can be recorded in the list B1, so that the list B1 records the plurality of home identities. In step 602, it can be queried whether the home identity A1 is in the list B1. If the home identity A1 is in the list B1, the plurality of home identities includes the home identity A1. If the home identity A1 is not in the list B1, the plurality of home identities does not include the home identity A1.

[0085] When the plurality of home identities includes the home identity A1, step 603 is performed, and the partition 311 corresponding to the home identity A1 is identified in the plurality of partitions.

[0086] In some embodiments, the correspondence between the partition and the home identity corresponding to the partition can be recorded in the list B2. For example, when the partition 311 is created based on the home identity A1, the correspondence between the home identity A1 and the partition 311 can be recorded in the list B2. In step 603, the partition corresponding to the home identity A1, i.e., the partition 311, can be identified based on the list B2.

[0087] In some embodiments, the data in different partitions of the same data table are respectively stored in different storage spaces, in other words, the plurality of partitions in the data table 310 and the plurality of storage spaces are in one-to-one correspondence, and the storage space corresponding to the partition is used to store data having the home identity corresponding to the partition. That is, the data in the partition is actually stored in the storage space corresponding to the partition. In step 603, the partition 311 corresponding to the home identity A1 is identified in the plurality of partitions, and then the storage space corresponding to the partition 311 is identified in the plurality of storage spaces. Exemplarily, the storage space can refer to a physical storage space composed of continuous physical storage addresses. Exemplarily, the storage space can be a physical page, a disk partition, a volume, etc.

[0088] In one example of the embodiment, the correspondence between the partition and the storage space can be recorded in the list B3. In step 603, the storage space corresponding to the partition 311 can be identified based on the list B3. The correspondence between the partition and the storage space can be the correspondence between the address of the partition and the address of the storage space. Identifying the storage space can be identifying the address of the storage space.

[0089] In some embodiments, the data in different partitions in the same data table are stored in different files. In other words, the multiple partitions in the data table 310 and the multiple files are one-to-one corresponding, and the file corresponding to a partition is used to store or record the data with the home identity corresponding to the partition. That is, the data in a partition is actually stored in the form of a file, and the data in different partitions are stored as different files. In step 603, the partition 311 corresponding to the home identity A1 is identified in the multiple partitions, and then the file corresponding to the partition 311 is identified in the multiple files.

[0090] In one example of this embodiment, the correspondence between the partition and the file can be recorded in the list B4. In step 603, the file corresponding to the partition 311 can be identified based on the list B4. In one example, the correspondence between the partition and the file can be the correspondence between the path of the partition and the file. Identifying the file can be identifying the path of the file. In one example, the correspondence between the partition and the file can be the correspondence between the handle of the partition and the file. Identifying the file can be identifying the handle of the file.

[0091] After step 603, step 604 can be performed to access the target data in the partition 311 based on the access request.

[0092] The target data is stored in the partition 311, and in step 604, the target data can be accessed in the local range of the data table 310 in the partition 311, without the need to access the target data in the global range of the data table 310. Therefore, only the partition 311 needs to be locked, without the need to lock the entire data table 310, so that the access to the target data does not affect the data access of other partitions. And the access to the target data only needs to cache the data in the partition 311, without the need to cache the data in other partitions, thereby saving the cache resource overhead and avoiding the cache thrashing problem that may occur when the access is frequent. In addition, through the pruning routing of the partition 311, the location of the target data (i.e. the partition 311) can be quickly located, thereby the efficiency of data access can be improved.

[0093] In some embodiments, as described above, the data with the home identity A1 is stored in the storage space corresponding to the partition 311, that is, the target data is stored in the storage space corresponding to the partition 311. In step 604, the target data can be accessed in the storage space corresponding to the partition 311 based on the access request. For example, according to the access request, the target data is inserted, read, modified or deleted in the storage space corresponding to the partition 311 based on the address of the storage space.

[0094] In some embodiments, as mentioned above, the data with the home identity A1 is stored in the file corresponding to the partition 311, i.e. the target data is stored in the file corresponding to the partition 311. In step 604, the target data can be accessed in the file corresponding to the partition 311 based on the access request. For example, according to the access request, the file corresponding to the partition 311 is located based on the path or handle of the file, and the target data is inserted, read, modified or deleted in the file.

[0095] In some embodiments, as mentioned above, the index can be created for the data in the partition 311. In step 604, the target data can be queried in the partition 311 based on the index of the partition 311. In the querying, only the index scan plan of the partition 311 is generated, thereby ensuring the correctness and efficiency of data access. In this way, the query efficiency of the target data can be improved. After the target data is queried, the target data can be accessed, e.g. inserted, read, modified or deleted, based on the access request.

[0096] Continuing to refer to FIG. 6, when the plurality of home identities does not include the home identity A1, step 605 can be performed to create the partition 313 corresponding to the home identity A1 in the data table 310.

[0097] The plurality of home identities does not include the home identity A1, which means that there is no partition corresponding to the home identity A1 in the data table 310. In this case, as shown in FIG. 7, the partition 313 corresponding to the home identity A1 can be created in the data table 310 to store the data with the home identity A1 in the partition 313.

[0098] In which, the partition 313 can be created according to the data routing result. Thus, the creation operation of the partition 313 does not block the data access of other partitions.

[0099] In some embodiments, the partition 313 can be created based on range division. For example, 1000-1100 rows in the data table 310 are divided into the partition 313. In some embodiments, the data table 310 is a logical table, a new physical table can be created and mapped to the data table 310, thereby obtaining the partition 313. In some embodiments, the data table 310 is a collection of physical tables, a new physical table can be added to the collection, and the new physical table is the partition 313. In some embodiments, the partition corresponds to a storage space, a new storage space can be divided in the memory, and the storage space is mapped to the partition 313, and the storage space is used to store the data corresponding to the partition 313. In some embodiments, the partition corresponds to a file, a new file can be created, and the file is mapped to the partition 313, and the file is used to store the data corresponding to the partition 313.

[0100] In some embodiments, the home identity A1 is a tenant identity. In step 605, a partition corresponding to the home identity A1 is created to store data of a tenant corresponding to the tenant identity in the partition.

[0101] For example, the partition 313 can be inserted in the data table 310 by an insert statement without separately executing a data table creation statement, such as a data definition language (DDL) statement.

[0102] After the partition 313 corresponding to the home identity is created in the data table 310, in step 606, target data can be accessed in the partition 313 based on an access request. The access request can be a write request or an insert request. In step 606, the target data is written or inserted in the partition 313.

[0103] In some embodiments, steps 605 and 606 are performed only when the plurality of home identities does not include the home identity A1 and the access request is a write request or an insert request. If the access request is a request other than a write request or an insert request, such as a read request, a modification request, or a deletion request, an empty result can be returned directly when the plurality of home identities does not include the home identity A1.

[0104] In some embodiments, when data corresponding to a home identity needs to be migrated, only data in a partition corresponding to the home identity needs to be migrated. For example, as shown in FIG. 8, the partition 311 can be migrated from the data table 310 to the data table 320, that is, the partition 311 is changed from a partition of the data table 310 to a partition of the data table 320. Since the partition is physically isolated from other partitions, when data is migrated, no row-by-row filtering copy is needed, and only data in the partition needs to be directly copied, so that data migration can be efficiently performed.

[0105] In summary, data of different home identities is stored in different partitions of a data table, and when data with a home identity is accessed, the data is accessed within a partition corresponding to the home identity, rather than within the entire data table, so that access to the data does not affect data access of other partitions, improving the concurrency of data access of the data table and improving the access performance of the data table.

[0106] Embodiments of the present application provide a database-based data processing apparatus. The database includes a data table, and the data table includes a plurality of partitions. The plurality of partitions and a plurality of home identities correspond to each other in a one-to-one manner, and the partitions are used to store data with a home identity corresponding to the partition. As shown in FIG. 9, the apparatus includes:

[0107] The receiving module 910 is configured to receive an access request, where the access request is used to request access to target data in the data table, and the target data has a target home identity;

[0108] The identifying module 920 is configured to identify, when the multiple home identities include the target home identity, a first partition corresponding to the target home identity in the multiple partitions;

[0109] The access module 930 is configured to access the target data in the first partition based on the access request.

[0110] In some embodiments, the first partition has an index, and the index is created based on data in the first partition;

[0111] The access module 930 is configured to:

[0112] query the target data in the first partition based on the index;

[0113] access the target data based on the access request.

[0114] In some embodiments, the multiple partitions and multiple storage spaces are in one-to-one correspondence, and a storage space corresponding to the partition is used to store data having a home identity corresponding to the partition;

[0115] The identifying module 920 is configured to identify, in the multiple storage spaces, a storage space corresponding to the first partition;

[0116] The access module 930 is configured to access the target data in the storage space corresponding to the first partition based on the access request.

[0117] In some embodiments, the multiple partitions and multiple files are in one-to-one correspondence, and a file corresponding to the partition is used to store data having a home identity corresponding to the partition;

[0118] The identifying module 920 is configured to identify, in the multiple files, a file corresponding to the first partition;

[0119] The access module 930 is configured to access the target data in the file corresponding to the first partition based on the access request.

[0120] In some embodiments, the apparatus further includes a creating module;

[0121] The creating module is configured to create, when the multiple home identities do not include the target home identity, a second partition corresponding to the target home identity in the data table;

[0122] The access module 930 is further configured to access the target data in the second partition based on the access request.

[0123] In some embodiments, different home identities in the plurality of home identities are different tenant identities, or different home identities in the plurality of home identities correspond to different data types.

[0124] The receiving module 910, the identification module 920, and the access module 930 can be implemented by software or by hardware. For example, the implementation of the receiving module 910 is described below. The implementation of the identification module 920 and the access module 930 can be similar to the implementation of the receiving module 910.

[0125] As an example of a software functional unit, the receiving module 910 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the receiving module 910 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes one data center or multiple data centers in close geographical proximity. Typically, one region includes multiple AZs.

[0126] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, one VPC is set up in one region. Interconnection between VPCs in the same region, and between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between VPCs.

[0127] As an example of a hardware functional unit, the receiving module 910 can include at least one computing device, such as a server or the like. Alternatively, the receiving module 910 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0128] The multiple computing devices included in the receiving module 910 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 910 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 910 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0129] It should be noted that, in other embodiments, the receiving module 910 can be configured to perform any of the steps of the method shown in FIG. 6, the identifying module 920 can be configured to perform any of the steps of the method shown in FIG. 6, and the accessing module 930 can be configured to perform any of the steps of the method shown in FIG. 6. The steps implemented by the receiving module 910, the identifying module 920, and the accessing module 930 can be specified as needed, and the overall function of the data storage device 900 can be implemented by the receiving module 910, the identifying module 920, and the accessing module 930 implementing different steps of the method shown in FIG. 6.

[0130] The present application also provides a computing device 1000. As shown in FIG. 10, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other through the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1000.

[0131] The bus 1002 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, or the like. For ease of representation, only one line is shown in FIG. 10, but it does not mean that there is only one bus or only one type of bus. The bus 1002 can include a path for transmitting information between various components (e.g., the memory 1006, the processor 1004, the communication interface 1008) of the computing device 1000.

[0132] The processor 1004 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), or the like.

[0133] The memory 1006 can include a volatile memory (e.g., a random access memory (RAM)), and can also include a non-volatile memory (e.g., a read-only memory (ROM), a floppy disk, a hard disk, or a solid state drive (SSD)).

[0134] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the aforementioned receiving module 910, the identifying module 920, and the accessing module 930, respectively, thereby implementing the method shown in FIG. 6. That is, the memory 1006 stores instructions for executing the method shown in FIG. 6.

[0135] The communication interface 1008 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, or the like, to implement communication between the computing device 1000 and other devices or communication networks.

[0136] The embodiments of the present application also provide a computing device cluster. The computing device cluster comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0137] As shown in FIG. 11, the computing device cluster comprises at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster can store the same instructions for performing the method shown in FIG. 6.

[0138] In some possible implementation manners, the memory 1006 in one or more computing devices 1000 in the computing device cluster can also respectively store partial instructions for performing the method shown in FIG. 6. In other words, the combination of one or more computing devices 1000 can collectively perform the instructions for performing the method shown in FIG. 6.

[0139] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, respectively for performing part of the functions of the apparatus 900. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more of the receiving module 910, the identifying module 920, and the accessing module 930.

[0140] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 12 shows one possible implementation manner. As shown in FIG. 12, two computing devices 1000A and 1000B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation manner, the memory 1006 in the computing device 1000A stores instructions for performing the functions of the receiving module 910. Meanwhile, the memory 1006 in the computing device 1000B stores instructions for performing the functions of the identifying module 920 and the accessing module 930.

[0141] It should be understood that the functions of the computing device 1000A shown in FIG. 12 can also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B can also be completed by multiple computing devices 1000.

[0142] The embodiment of the present application further provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection mode of the computing device cluster described with reference to FIG. 11 and FIG. 12. The difference is that the same instruction for executing the method shown in FIG. 6 can be stored in the memory 1006 of one or more computing devices 1000 in the computing device cluster.

[0143] In some possible implementation manners, part of the instruction for executing the method shown in FIG. 6 can also be stored in the memory 1006 of one or more computing devices 1000 in the computing device cluster respectively. In other words, the combination of one or more computing devices 1000 can collectively execute the instruction for executing the method shown in FIG. 6.

[0144] The embodiment of the present application further provides a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to execute the method shown in FIG. 6.

[0145] The embodiment of the present application further provides a computer readable storage medium. The computer readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like. The computer readable storage medium contains instructions, which instruct the computing device to execute the method shown in FIG. 6.

[0146] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A database-based data processing method, characterized by, The database comprises a data table, the data table comprises a plurality of partitions, the plurality of partitions and a plurality of home identifiers are in one-to-one correspondence, and the partitions are used for storing data with the home identifiers corresponding to the partitions; the method comprises: receiving an access request, the access request being used for requesting to access target data in the data table, the target data having a target home identifier; when the plurality of home identifiers comprises the target home identifier, identifying a first partition corresponding to the target home identifier in the plurality of partitions; based on the access request, accessing the target data in the first partition.

2. The method of claim 1, wherein, The first partition has an index, and the index is created based on data in the first partition; the accessing the target data in the first partition based on the access request comprises: querying the target data in the first partition based on the index; accessing the target data based on the access request.

3. The method according to claim 1 or 2, characterized in that, The plurality of partitions and a plurality of storage spaces are in one-to-one correspondence, and the storage space corresponding to the partition is used for storing data with the home identifier corresponding to the partition; the identifying the first partition corresponding to the target home identifier in the plurality of partitions comprises: identifying a storage space corresponding to the first partition in the plurality of storage spaces; the accessing the target data in the first partition based on the access request comprises: accessing the target data in the storage space corresponding to the first partition based on the access request.

4. The method according to claim 1 or 2, characterized in that, The plurality of partitions and a plurality of files are in one-to-one correspondence, and the file corresponding to the partition is used for storing data with the home identifier corresponding to the partition; the identifying the first partition corresponding to the target home identifier comprises: identifying a file corresponding to the first partition in the plurality of files; the accessing the target data in the first partition based on the access request comprises: accessing the target data in the file corresponding to the first partition based on the access request.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: when the plurality of home identifiers does not comprise the target home identifier, creating a second partition corresponding to the target home identifier in the data table; based on the access request, accessing the target data in the second partition.

6. The method according to any one of claims 1-5, characterized in that, Different home identifiers in the plurality of home identifiers are respectively identifiers of different tenants, or different home identifiers in the plurality of home identifiers respectively correspond to different data types.

7. A database-based data processing apparatus, characterized by comprising: The database comprises a data table, the data table comprises a plurality of partitions, the plurality of partitions and a plurality of home identifiers are in one-to-one correspondence, and the partitions are used for storing data with the home identifiers corresponding to the partitions; the device comprises: a receiving module, configured to receive an access request, the access request being used for requesting to access target data in the data table, the target data having a target home identifier; an identifying module, configured to, when the plurality of home identifiers comprises the target home identifier, identify a first partition corresponding to the target home identifier in the plurality of partitions; an accessing module, configured to, based on the access request, access the target data in the first partition.

8. The apparatus of claim 7, wherein, The first partition has an index, the index is created based on data in the first partition; The access module is configured to: query the target data in the first partition based on the index; access the target data based on the access request.

9. The apparatus of claim 7 or 8, wherein, The plurality of partitions and the plurality of storage spaces are in one-to-one correspondence, and the storage space corresponding to the partition is used to store data having a home identity corresponding to the partition; The identification module is configured to identify, in the plurality of storage spaces, a storage space corresponding to the first partition; The access module is configured to access the target data in the storage space corresponding to the first partition based on the access request.

10. The apparatus of claim 7 or 8, wherein, The plurality of partitions and the plurality of files are in one-to-one correspondence, and the file corresponding to the partition is used to store data having a home identity corresponding to the partition; The identification module is configured to identify, in the plurality of files, a file corresponding to the first partition; The access module is configured to access the target data in the file corresponding to the first partition based on the access request.

11. The apparatus of any one of claims 7-10, wherein, The apparatus further includes a creation module; The creation module is configured to create a second partition corresponding to the target home identity in the data table when the plurality of home identities does not include the target home identity; The access module is further configured to access the target data in the second partition based on the access request.

12. The apparatus of any one of claims 7-11, wherein, Different home identities in the plurality of home identities are different tenant identities, or different home identities in the plurality of home identities correspond to different data types.

13. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-6.

15. A computer program product comprising instructions, characterized in that, The instructions, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method of any one of claims 1-6.

Images