Data processing method based on distributed database and related equipment

Abstract

The invention provides a data processing method based on a distributed database and related equipment. A data table in the distributed database is divided into a plurality of partitions; a plurality of physical partition objects corresponding to the partitions and data structure information corresponding to each physical partition object are maintained in a storage layer of the distributed database; the method comprises the steps of obtaining a data operation request for a target data table; analyzing the data operation request to determine a target partition to be operated from a plurality of partitions contained in the target data table; acquiring maintained data structure information corresponding to a target physical partition object corresponding to the target partition; and performing optimization processing on an execution plan generated based on the data operation request according to the data structure information, and performing data operation on data stored in the target physical partition object based on the optimized execution plan.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a data processing method based on a distributed database and related equipment. Background Art

[0002] Different from traditional stand-alone database systems, distributed database systems use data sharding technology to achieve database scalability. Specifically, the data in a data table can be horizontally split into multiple shards according to certain rules. Each shard can be called a table partition, or partition for short. Different partitions of the same table and different partitions of different tables can be distributed to multiple machines according to certain strategies to achieve horizontal expansion of service capabilities and storage capacity. Among them, the partition is the smallest unit for data migration in a distributed database. When the system service capabilities or storage capacity need to be expanded, it can be achieved by migrating the partition to other service nodes.

[0003] In the storage layer of the distributed database, multiple physical partition objects corresponding to multiple partitions are maintained, and each physical partition object stores the data of the corresponding partition. Among them, a non-partitioned table can correspond to a physical partition object; each first-level partition of a first-level partitioned table can correspond to a physical partition object; each second-level partition of a second-level partitioned table can correspond to a physical partition object.

[0004] Based on this, in a distributed database, how to efficiently perform data operations on data stored in different physical partition objects is an urgent problem to be solved. Summary of the invention

[0005] In view of this, one or more embodiments of the present specification provide a data processing method and related devices based on a distributed database.

[0006] In a first aspect, the present specification provides a data processing method based on a distributed database, wherein a data table in the distributed database is divided into a plurality of partitions; a storage layer of the distributed database maintains a plurality of physical partition objects corresponding to the plurality of partitions, and data structure information corresponding to each physical partition object;

[0007] The method comprises:

[0008] Get the data operation request for the target data table;

[0009] Parsing the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table;

[0010] Acquire the maintained data structure information corresponding to the target physical partition object corresponding to the target partition;

[0011] According to the data structure information, an execution plan generated based on the data operation request is optimized, and based on the optimized execution plan, a data operation is performed on the data stored in the target physical partition object.

[0012] In a second aspect, the present specification provides a data processing device based on a distributed database, wherein a data table in the distributed database is divided into a plurality of partitions; a storage layer of the distributed database maintains a plurality of physical partition objects corresponding to the plurality of partitions, and data structure information corresponding to each physical partition object;

[0013] The device comprises:

[0014] A first acquisition unit, used to acquire a data operation request for a target data table;

[0015] a determination unit, configured to parse the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table;

[0016] A second acquisition unit, configured to acquire the maintained data structure information corresponding to the target physical partition object corresponding to the target partition;

[0017] An execution unit is used to optimize an execution plan generated based on the data operation request according to the data structure information, and perform data operations on the data stored in the target physical partition object based on the optimized execution plan.

[0018] Accordingly, the present specification also provides a computing device, comprising: a memory and a processor; the memory stores computer programs / instructions that can be executed by the processor; when the processor executes the computer program / instructions, it executes the data processing method based on the distributed database described in the first aspect above.

[0019] Accordingly, the present specification also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the data processing method based on a distributed database as described in the first aspect above is executed.

[0020] Accordingly, the present specification also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the data processing method based on a distributed database as described in the first aspect above is executed.

[0021] In summary, the present application is applied to a distributed database, in which a data table is divided into multiple partitions; a storage layer of the distributed database maintains multiple physical partition objects corresponding to the multiple partitions, and data structure information corresponding to each physical partition object. In an illustrated embodiment, a data operation request for a target data table can be obtained first. Then, the data operation request is parsed to determine the target partition to be operated from the multiple partitions contained in the target data table. Then, the storage layer obtains the data structure information corresponding to the target physical partition object corresponding to the target partition. Further, the execution plan generated based on the data operation request can be optimized according to the data structure information, and the data operation is performed on the data stored in the target physical partition object based on the optimized execution plan. In this way, the present application achieves the tight coupling of the physical partition objects in the storage layer with the schema information by maintaining multiple physical partition objects corresponding to the multiple partitions of the data table one by one, and data structure information (such as schema information) corresponding to each physical partition object one by one in the storage layer of the distributed database. In this way, the storage layer has mastered the schema information, which is equivalent to mastering the data rules in the partitions corresponding to each physical partition object. It can then optimize the execution plan generated based on the current data operation request to perform efficient data operations on the data stored in the physical partition object, improve the performance of the distributed database system, and ensure the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of a system architecture of a distributed database provided by an exemplary embodiment;

[0023] Figure 2 It is a flowchart of a data processing method based on a distributed database provided by an exemplary embodiment;

[0024] Figure 3 It is a schematic diagram of a tight coupling between a physical partition object and schema information provided by an exemplary embodiment;

[0025] Figure 4 It is a structural schematic diagram of a data processing device based on a distributed database provided by an exemplary embodiment;

[0026] Figure 5 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0030] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] This specification provides a technical solution that tightly couples multiple physical partition objects in the storage layer with corresponding data structure information (such as schema information). In this way, the storage layer masters the schema information, which is equivalent to mastering the data rules, and can then optimize the execution plan of the current data operation to perform efficient data operations on the data stored in the physical partition objects.

[0032] In implementation, the present application is applied to a distributed database, in which a data table is divided into multiple partitions; a plurality of physical partition objects corresponding to the multiple partitions, and data structure information corresponding to each physical partition object are maintained in the storage layer of the distributed database. In one illustrated embodiment, a data operation request for a target data table may be first obtained. Then, the data operation request is parsed to determine the target partition to be operated from the multiple partitions contained in the target data table. Then, the storage layer obtains the data structure information corresponding to the target physical partition object maintained corresponding to the target partition. Furthermore, the execution plan generated based on the data operation request may be optimized based on the data structure information, and data operations may be performed on the data stored in the target physical partition object based on the optimized execution plan.

[0033] In the above technical solution, the present application realizes the tight coupling of the physical partition objects in the storage layer with the schema information by maintaining multiple physical partition objects corresponding to multiple partitions of the data table in the storage layer of the distributed database, and data structure information (such as schema information) corresponding to each physical partition object. In this way, the storage layer has mastered the schema information, which is equivalent to mastering the data rules in the partitions corresponding to each physical partition object, and can then optimize the execution plan generated based on the current data operation request to perform efficient data operations (such as data read and write operations, etc.) on the data stored in the physical partition objects, improve the performance of the distributed database system, and ensure the user experience.

[0034] See also Figure 1 , Figure 1 This is a schematic diagram of a system architecture based on a distributed database provided by an exemplary embodiment. One or more embodiments provided in this specification may be Figure 1 The present invention is specifically implemented in the system architecture shown or a similar system architecture.

[0035] like Figure 1 As shown, users can efficiently access the distributed database through a proxy module (proxy) connected to the distributed database, i.e., database access middleware, to perform corresponding data operations on the data therein, such as read and write operations or data query operations, etc. This specification does not make specific limitations on this.

[0036] like Figure 1 As shown, the distributed database system may include multiple availability zones, where an availability zone refers to a physical area within a data center that contains resources and services of a cloud service provider or a large distributed system. Each availability zone has an independent power supply, cooling system, and network connection to reduce the risk of single point failures and provide physical isolation. Figure 1 As shown, each availability zone may include multiple service nodes (servers), and each service node may be composed of one or more devices (such as servers), which is not specifically limited in this specification.

[0037] like Figure 1 As shown, the distributed database can also support a multi-tenant architecture, for example, including Figure 1 Tenant 1 and Tenant 2, etc., each tenant can have a database instance exclusively. The multi-tenant architecture allows different tenants to run on the same physical cluster, and the data between tenants can be completely isolated. In this way, the distributed database in this application can efficiently utilize resources and flexibly adjust resource allocation according to service requirements.

[0038] like Figure 1 As shown in the figure, the data table in the distributed database is divided into multiple partitions, and each partition is stored in multiple availability zones in the form of multiple copies to achieve horizontal expansion of service capabilities and storage capacity. Distributed databases organize data in partitions, which are the smallest unit of data migration. When the system service capabilities or storage capacity need to be expanded, it can be achieved by migrating the partitions to other service nodes.

[0039] For example, Figure 1 As shown, the data table of tenant 1 is divided into four partitions, namely partition P1, partition P2, partition P3 and partition P4. The multiple partitions can be distributed on multiple service nodes. For example, partition P1 and partition P2 are distributed on one service node, and partition P3 and partition P4 are distributed on another service node.

[0040] Furthermore, each of the partitions P1, P2, P3, and P4 can be stored in multiple availability zones in the form of multiple copies, where the multiple copies can include a primary copy and a secondary copy. Figure 1 As shown, the primary copy of partition P1 is stored in availability zone 1, the secondary copy of partition P1 is stored in availability zone 2, and another secondary copy of partition P1 is stored in availability zone 3; Figure 1 As shown, the slave copy of partition P2 is stored in availability zone 1, the master copy of partition P2 is stored in availability zone 2, and another slave copy of partition P2 is stored in availability zone 3; Figure 1 As shown, the slave copy of partition P3 is stored in availability zone 1, the master copy of partition P3 is stored in availability zone 2, another slave copy of partition P3 is stored in availability zone 3, and so on. We will not go into details here. For details, please refer to Figure 1 The system architecture is shown.

[0041] As mentioned above, the distributed database system achieves system scalability and high availability through data sharding and multi-copy replication technology. The horizontal expansion of system service capabilities and storage capacity is achieved by migrating data shards (i.e. partitions) between different machines. The high availability of system services is achieved by replicating each partition to multiple availability zones through multi-copy replication technology. Furthermore, by adjusting the availability zone configuration of different replicas, different levels of disaster recovery capabilities can be achieved, including: machine-level disaster recovery, rack-level disaster recovery, computer room-level disaster recovery, and regional-level disaster recovery, etc., ensuring data security and continuous operation of the system.

[0042] In addition, the partition design has the following four advantages:

[0043] (1) Data management: Partitioning makes data in large tables easier to manage because operations can be performed on a single partition rather than the entire table.

[0044] (2) Data locality: Related data can be stored in the same or adjacent partitions, thereby reducing data movement across partitions or nodes during queries.

[0045] (3) Parallel processing: Different partitions can process queries and transactions in parallel on different service nodes, thereby improving performance.

[0046] (4) Independence: Each partition can be backed up, restored, or migrated independently of other partitions, which helps maintain and troubleshoot the entire system.

[0047] It should be noted that Figure 1 The system architecture shown is only for illustrative purposes. In some possible implementations, the distributed database system may include Figure 1 The present specification does not specifically limit the structures shown in the figure to more or less.

[0048] See also Figure 2 , Figure 2 is a flowchart of a data processing method based on a distributed database provided by an exemplary embodiment. The method can be applied to Figure 1 The distributed database shown in Figure 1. Figure 2 As shown, the method may specifically include the following steps S201 to S204.

[0049] Step S201, obtaining a data operation request for a target data table in a distributed database; wherein the data table in the distributed database is divided into a plurality of partitions; a storage layer of the distributed database maintains a plurality of physical partition objects corresponding to the plurality of partitions, and data structure information corresponding to each physical partition object.

[0050] As described above, the data table in the distributed database is divided into multiple partitions, and the storage layer of the distributed database can maintain multiple physical partition objects corresponding to the multiple partitions. Furthermore, the storage layer can also maintain data structure information corresponding to each physical partition object, such as schema information. Schema refers to the logical structure of the database, including database metadata such as tables, columns, data types, indexes and constraints, etc., and this specification does not make specific limitations on this.

[0051] In an illustrated embodiment, the schema information corresponding to each physical partition object may specifically include at least one of the following information: the name of the data table corresponding to each physical partition object, the index structure of the data table, the field name and field type in the data table, and the partition name and partition level (for example, a first-level partition or a second-level partition, etc.) of the partition corresponding to each physical partition object, etc. This specification does not make any specific limitations on this.

[0052] In one illustrated embodiment, see Figure 3 , Figure 3 It is a schematic diagram of a tight coupling between a physical partition object and schema information provided by an exemplary embodiment.

[0053] like Figure 3 As shown, data table T1 is a non-partitioned table, corresponding to a physical partition object T1 in the storage layer, and the physical partition object T1 is tightly coupled with the schema information of data table T1.

[0054] like Figure 3 As shown, data table T2 is a first-level partition table, which contains two first-level partitions, namely partition P0 and partition P1. Partition P0 corresponds to the physical partition object T2P0 in the storage layer, and the physical partition object T2P0 is tightly coupled with the schema information of partition P0; partition P1 corresponds to the physical partition object T2P1 in the storage layer, and the physical partition object T2P1 is tightly coupled with the schema information of partition P1.

[0055] like Figure 3As shown, data table T3 is a secondary partition table, which includes three secondary partitions, namely partition P0-SP0, partition P0-SP1 and partition P1-SP0. Partition P0-SP0 corresponds to the physical partition object T3P0SP0 in the storage layer, and the physical partition object T3P0SP0 is tightly coupled with the schema information of partition P0-SP0; partition P0-SP1 corresponds to the physical partition object T3P0SP1 in the storage layer, and the physical partition object T3P0SP1 is tightly coupled with the schema information of partition P0-SP1; partition P1-SP0 corresponds to the physical partition object T3P1SP0 in the storage layer, and the physical partition object T3P1SP0 is tightly coupled with the schema information of partition P1-SP0.

[0056] As described above, each partition is stored in multiple availability zones in the form of multiple copies. Therefore, each physical partition object may contain multiple copies of the partition corresponding to the physical partition object. Figure 3 For example, the physical partition object T1 may contain multiple copies of the data table T1; the physical partition object T2P0 may contain multiple copies of the partition P0, the physical partition object T2P1 may contain multiple copies of the partition P1; the physical partition object T3P0SP0 may contain multiple copies of the partitions P0-SP0, and so on.

[0057] From the perspective of schema, each physical partition object can determine the identifier of the physical partition object through the schema information corresponding to it. Specifically, a triple can be uniquely determined by the schema information corresponding to the physical partition object. Exemplarily, the triple can include the identifier of the data table corresponding to the physical partition object, the first-level partition identifier and the second-level partition identifier, which can be recorded as {table_id, part_id, subpart_id}. Furthermore, the identifier of the physical partition object can be generated by encoding the triple, such as the unique ID of the physical partition object: PartitionKey. In an illustrated embodiment, the PartitionKey can be persisted in the storage metadata of the physical partition object, and the storage layer can manage the physical partition object based on the PartitionKey. In addition, in an illustrated embodiment, the storage metadata can also persist information such as the column ID and column type of the partition, and this information can all be derived from the Schema information corresponding to the physical partition object.

[0058] In one illustrated embodiment, metadata tables related to physical partition objects can also be stored and accessed based on PartitionKey as the primary key. For example, the replica information table (meta table) of the physical partition object records the location information, status information, etc. of the partition replica; the column checksum information table also uses PartitionKey to identify the information of a partition, etc. This specification does not make specific limitations on this.

[0059] In an illustrated embodiment, a user may initiate a data operation request for a target data table to a distributed database, and correspondingly, the distributed database may receive the data operation request for the target data table initiated by the user. Exemplarily, the target data table may be a primary partition table or a secondary partition table in a distributed database, which is not specifically limited in this specification. Exemplarily, the data operation may be a read operation, a write operation, or a data query operation, etc., which is not specifically limited in this specification.

[0060] In one illustrated embodiment, a user may Figure 1 The proxy shown initiates a data operation request for a target data table to the distributed database. Accordingly, after receiving the data operation request for the target data table, the proxy can choose to send the operation request to a suitable service node for processing. This specification does not make any specific limitations on this.

[0061] Step S202: parsing the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table.

[0062] Furthermore, the distributed database may parse the received data operation request to determine a target partition to be operated from multiple partitions included in the target data table, and further, may determine one or more target data records to be operated from the target partition.

[0063] In an illustrated implementation manner, the data operation request may be parsed by a query engine in a distributed database to determine the target partition and target data record to be operated.

[0064] In an illustrated embodiment, the target data table may be a primary partition table, and the target partition may be one of multiple primary partitions included in the target data table. In an illustrated embodiment, the target data table may also be a secondary partition table, and the target partition may be one of multiple secondary partitions included in the target data table, which is not specifically limited in this specification.

[0065] Step S203: obtaining the maintained data structure information corresponding to the target physical partition object corresponding to the target partition.

[0066] Furthermore, after determining the target partition and target data record targeted by this data operation request, the target physical partition object corresponding to the target partition can be determined, and the data structure information corresponding to the target physical partition object maintained in the storage layer, namely, the schema information, can be obtained.

[0067] Step S204: optimizing the execution plan generated based on the data operation request according to the data structure information, and performing data operations on the data stored in the target physical partition object based on the optimized execution plan.

[0068] Furthermore, the execution plan generated based on the current data operation request may be optimized according to the acquired schema information corresponding to the target physical partition object.

[0069] Furthermore, based on the optimized execution plan, data operations may be performed on the data stored in the target physical partition object, and specifically, data operations may be performed on the target data records stored in the target physical partition object.

[0070] In an illustrated embodiment, if the current data operation request also involves the modification of schema information, the schema information corresponding to the target physical partition object maintained in the storage layer may be modified first, and then the data operation may be further performed on the target data record stored in the target physical partition object. This specification does not make specific limitations on this. Exemplarily, the modification of schema information may include various situations such as modifying the data type of a field in a table or modifying the field name. This specification does not make specific limitations on this. Exemplarily, assuming that the current data operation request involves modifying the data type of a field in a table, the data type of a field recorded in the schema information corresponding to the target physical partition object maintained in the storage layer may be modified from integer type to floating point type. After the modification is completed, the data operation may be further performed on the target data record stored in the target physical partition object, such as writing floating point data in the field of the target data record, etc. This specification does not make specific limitations on this.

[0071] In addition, it should be noted that this specification does not specifically limit the specific implementation method of optimizing the execution plan based on schema information.

[0072] In one illustrated embodiment, after the storage layer has mastered the schema information, it can adopt a data encoding rule (such as Huffman coding, etc.) that matches the data in the target partition to be operated, and optimize the execution plan of the current data operation (such as data reading and writing operations, etc.) based on the data encoding rule, such as performing data compression processing, etc., and for example, it can optimize analytical queries through columnar storage, or use a specific index structure to improve data query performance, and any other possible optimization processing methods, which are not specifically limited in this specification.

[0073] As described above, since each partition is stored in multiple availability zones in the form of multiple copies, when performing data operations on the target data records stored in the target physical partition object, it can specifically include: performing data operations on the target data records stored in multiple copies of the target partition contained in the target physical partition object respectively to ensure data synchronization of multiple copies.

[0074] In an illustrated embodiment, as described above, the storage layer also maintains an identifier of each physical partition object (i.e., a unique ID: PartitionKey). Therefore, based on the identifier of the maintained target physical partition object, multiple copies of the target partition corresponding to the target physical partition object can be found in multiple copies of multiple partitions stored in multiple availability zones; further, data operations can be performed on the target data records stored in the multiple copies of the target partition, respectively.

[0075] In an embodiment shown, the distributed database in the present application also couples data sharding (i.e., partitions) and log streams in implementation, i.e., a log stream is bound to each partition. The log stream of each partition may contain multiple transaction logs generated in sequence related to the partition.

[0076] Based on this, when performing data operations on target data records stored in multiple copies of the target partition contained in the target physical partition object, it can specifically include: first writing the data operation for the target data record into the log file corresponding to the target partition, and generating a corresponding transaction log; further, in response to the successful submission of the transaction containing the data operation, the data operation can be performed on the target data records stored in multiple copies of the target partition contained in the target physical partition object according to the transaction log contained in the log stream corresponding to the target partition. In an illustrated embodiment, the data operation can be performed on the target data record stored in the primary copy of the target partition contained in the target physical partition object first; and after the data operation on the primary copy is successful, the data operation can be synchronously performed on the target data record stored in the secondary copy of the target partition contained in the target physical partition object to ensure data synchronization of multiple copies.

[0077] In one illustrated embodiment, the transaction engine in the distributed database may specifically write the data operation for the target data record into a log file corresponding to the target partition and generate a corresponding transaction log, etc. This specification does not specifically limit this.

[0078] In one illustrated implementation, a storage engine in a distributed database may specifically perform data operations on target data records stored in multiple copies of a target partition included in the target physical partition object, etc. This specification does not specifically limit this.

[0079] In an illustrated implementation, multiple copies of the same partition can achieve data synchronization through a preset consistency protocol. It should be noted that the present application does not specifically limit the specific type of the consistency protocol. In an illustrated implementation, the present application can synchronize multiple copies of the partition log through the Paxos protocol to ensure that data consistency and high availability of the distributed database system can be maintained even in the event of partial failures. Please also refer to Figure 1 ,like Figure 1 As shown, multiple replicas of a partition (for example, partition P1 contained in the data table of tenant 2) can be used as a Paxosgroup (combination), and multiple replicas in the combination can achieve data synchronization through the Paxos protocol to ensure data consistency.

[0080] In one illustrated embodiment, when the distributed database system detects that the number of replicas of a partition is lower than a predetermined preset threshold, the distributed database will automatically trigger an operation to replenish the replicas and copy the data to a new machine. This process further enhances the disaster recovery capability and data persistence of the distributed database system.

[0081] In summary, through the above-mentioned data sharding and multi-copy replication mechanism, the distributed database in this application can ensure strong data consistency, while providing failover and disaster recovery capabilities, thereby meeting the strict requirements of key service systems for databases. These characteristics of distributed databases become particularly important, especially in high concurrency and big data scenarios, because they ensure the continuity of system services and the integrity of data. In addition, the distributed database system in this application couples data sharding (i.e., partitioning) and log streams in implementation, that is, a log stream is bound to each data shard, achieving high availability at the data shard level, which brings great flexibility to the system's functions, and has the following 4 advantages:

[0082] (1) Customized replica distribution and disaster recovery capabilities: Users can adjust the replica distribution and disaster recovery configuration at different levels (such as database level, table level, or partition level) according to service needs. For example, for some critical data, users may choose to store multiple replicas in more availability zones to enhance disaster recovery capabilities; while for less critical data, they may only store them in a few availability zones to save resources. In addition, the Primary Zone configuration allows users to specify the primary write area for data to optimize write performance.

[0083] (2) Data distribution heterogeneity: In different zones, the system can support different numbers of service nodes to cope with different load requirements. This heterogeneous distribution capability allows users to customize resource planning for different application scenarios and performance requirements. The system can automatically achieve load balancing to ensure the performance and efficiency of data access.

[0084] (3) Fine-grained operation and maintenance: Users can perform partition-level replica management, including repairing, deleting, or rebuilding damaged replica data. For hot data partitions with very high access volume, the system allows users to isolate these hot partitions and deploy them on dedicated servers to reduce the impact of hot issues on other partitions and services, so that such impact can be limited to the same zone.

[0085] (4) Fault isolation: In the multi-replica log synchronization of the Paxos protocol, if a minority replica fails, it can be ensured that the failure only affects the corresponding partition replica. Such fault isolation can reduce the scope and impact caused by single point failures, thereby improving the stability and reliability of the entire system.

[0086] In summary, the storage layer of the distributed database of the present application maintains multiple schema information corresponding to multiple physical partition objects, realizing tight coupling between storage and schema information, which can bring the following beneficial effects:

[0087] (1) Data consistency: Since any change to the schema may directly affect the storage layer, the design of tightly coupling storage and schema helps ensure the consistency of the data model.

[0088] (2) Simplified design: Tight coupling of storage and schema can simplify the architectural design of distributed databases, making distributed database systems easier to understand and maintain.

[0089] (3) Performance optimization: With the tight coupling of storage and schema information, the storage layer has access to schema information, which is equivalent to understanding the data patterns in the data table. Therefore, the execution plan of data read and write operations can be optimized, improving the performance of the distributed database and ensuring the user experience.

[0090] In addition, the storage and schema tightly coupled design means that the storage engine of the distributed database is designed for a specific schema, which can provide optimization for massive data and high concurrency. At the same time, multi-tenant architectures may also benefit from this storage and schema tightly coupled design, because it can provide strong isolation for different tenants while maintaining efficient resource utilization.

[0091] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a data processing device based on a distributed database. Figure 4 , Figure 4 is a schematic diagram of a data processing device based on a distributed database provided by an exemplary embodiment. The device 40 can be applied to Figure 1 The distributed database shown in the figure has a data table divided into multiple partitions; the storage layer of the distributed database maintains multiple physical partition objects corresponding to the multiple partitions, as well as data structure information corresponding to each physical partition object. Figure 4 As shown, the device 40 includes:

[0092] A first acquisition unit 401 is used to acquire a data operation request for a target data table;

[0093] A determination unit 402 is used to parse the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table;

[0094] A second acquisition unit 403 is used to acquire the maintained data structure information corresponding to the target physical partition object corresponding to the target partition;

[0095] The execution unit 404 is used to optimize the execution plan generated based on the data operation request according to the data structure information, and perform data operations on the data stored in the target physical partition object based on the optimized execution plan.

[0096] In an illustrated embodiment, the data structure information includes schema information.

[0097] In an illustrated implementation manner, the determining unit 402 is specifically configured to:

[0098] Parsing the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table, and further determining a target data record to be operated from the target partition;

[0099] The execution unit 404 is specifically used for:

[0100] A data operation is performed on the target data record stored in the target physical partition object.

[0101] In an illustrated embodiment, each of the plurality of partitions is stored in a plurality of devices in the form of multiple copies; each physical partition object includes multiple copies of the partition corresponding to the physical partition object;

[0102] The execution unit 404 is specifically used for:

[0103] Data operations are respectively performed on the target data records stored in multiple copies of the target partition included in the target physical partition object.

[0104] In an illustrated embodiment, each partition corresponds to a log stream, and the log stream includes a plurality of transaction logs generated in sequence; the execution unit 404 is specifically configured to:

[0105] Writing the data operation for the target data record into a log file corresponding to the target partition, and generating a corresponding transaction log;

[0106] In response to successful submission of a transaction including the data operation, data operations are respectively performed on the target data records stored in multiple copies of the target partition included in the target physical partition object according to the transaction log included in the log stream corresponding to the target partition.

[0107] In an illustrated embodiment, the multiple replicas include a master replica and a slave replica; the execution unit 404 is specifically configured to:

[0108] performing data operations on the target data records stored in the primary copy of the target partition contained in the target physical partition object; and,

[0109] After the data operation on the primary copy is successful, the data operation is synchronously performed on the target data record stored in the secondary copy of the target partition contained in the target physical partition object.

[0110] In an illustrated embodiment, the distributed database includes multiple availability zones, each availability zone includes at least one device; each partition is stored in the multiple availability zones in the form of multiple copies; the storage layer also maintains the identifier of each physical partition object;

[0111] The execution unit 404 is specifically used for:

[0112] Based on the maintained identifier of the target physical partition object, searching for multiple replicas of the target partition corresponding to the target physical partition object from multiple replicas of the multiple partitions stored in the multiple availability zones;

[0113] Data operations are performed on the target data records stored in multiple copies of the target partition respectively.

[0114] In an illustrated embodiment, the identifier of each physical partition object is an identifier generated based on schema information corresponding to the physical partition object.

[0115] In an illustrated embodiment, the identifier of each physical partition object is an identifier generated based on a triple corresponding to the physical partition object, and the triple is a triple generated based on the schema information corresponding to the physical partition object; the triple includes the identifier of the data table corresponding to the physical partition object, the first-level partition identifier, and the second-level partition identifier.

[0116] In an illustrated embodiment, the execution unit 404 is specifically configured to:

[0117] Determining, according to the schema information, a data encoding rule that matches the data in the target partition targeted by the data operation request;

[0118] According to the data encoding rule, an execution plan generated based on the data operation request is optimized.

[0119] In one illustrated embodiment, the schema information corresponding to each physical partition object includes at least one of the following information: the name of the data table corresponding to each physical partition object, the index structure of the data table, the field name and field type in the data table, and the partition name and partition level of the partition corresponding to each physical partition object.

[0120] The implementation process of the functions and effects of each unit in the above-mentioned device 40 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 40 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0121] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0122] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0123] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Figure 5 , Figure 5 It is a structural diagram of a computing device provided by an exemplary embodiment. Figure 5 The computing device shown may be the above Figure 1 Any service node in the system architecture of the distributed database shown in the figure; the data table in the distributed database is divided into multiple partitions; the storage layer of the distributed database maintains multiple physical partition objects corresponding to the multiple partitions, and data structure information corresponding to each physical partition object. Figure 5 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 5As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the data processing method based on the distributed database in the embodiment of this specification, including: obtaining a data operation request for a target data table; parsing the data operation request to determine the target partition to be operated from the multiple partitions contained in the target data table; obtaining the data structure information corresponding to the target physical partition object corresponding to the target partition; according to the data structure information, optimizing the execution plan generated based on the data operation request, and based on the optimized execution plan, performing data operations on the data stored in the target physical partition object, and so on.

[0124] For a detailed description of each step of the above-mentioned data processing method based on a distributed database, please refer to the previous content and will not be repeated here.

[0125] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when these computer programs are executed by a processor, the various steps of the data processing method based on a distributed database in the embodiment of this specification are executed. Please refer to the description of the above embodiment for details, which will not be repeated here.

[0126] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0127] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0128] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0129] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0130] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A data processing method based on a distributed database, characterized in that: The data table in the distributed database is divided into a plurality of partitions; a plurality of physical partition objects corresponding to the plurality of partitions and data structure information corresponding to each physical partition object are maintained in the storage layer of the distributed database; The method comprises: Get the data operation request for the target data table; Parsing the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table; Acquire the maintained data structure information corresponding to the target physical partition object corresponding to the target partition; According to the data structure information, an execution plan generated based on the data operation request is optimized, and based on the optimized execution plan, a data operation is performed on the data stored in the target physical partition object.

2. The method according to claim 1, characterized in that The data structure information includes schema information.

3. The method according to claim 2, characterized in that The parsing of the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table includes: Parsing the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table, and further determining a target data record to be operated from the target partition; The performing of data operations on the data stored in the target physical partition object comprises: A data operation is performed on the target data record stored in the target physical partition object.

4. The method according to claim 3, characterized in that Each of the plurality of partitions is stored in a plurality of devices in the form of multiple copies; each physical partition object includes multiple copies of the partition corresponding to the physical partition object; The performing of data operations on the target data records stored in the target physical partition object comprises: Data operations are respectively performed on the target data records stored in multiple copies of the target partition included in the target physical partition object.

5. The method according to claim 4, characterized in that Each partition corresponds to a log stream, and the log stream includes multiple transaction logs generated in sequence; performing data operations on the target data records stored in multiple copies of the target partition included in the target physical partition object respectively includes: Writing the data operation for the target data record into a log file corresponding to the target partition, and generating a corresponding transaction log; In response to successful submission of a transaction including the data operation, data operations are respectively performed on the target data records stored in multiple copies of the target partition included in the target physical partition object according to the transaction log included in the log stream corresponding to the target partition.

6. The method according to claim 5, characterized in that The multiple copies include a master copy and a slave copy; the target data records stored in the multiple copies of the target partition included in the target physical partition object are respectively subjected to data operations, including: performing data operations on the target data records stored in the primary copy of the target partition contained in the target physical partition object; and, After the data operation on the primary copy is successful, the data operation is synchronously performed on the target data record stored in the secondary copy of the target partition contained in the target physical partition object.

7. The method according to claim 4, characterized in that The distributed database includes multiple availability zones, each of which includes at least one device; each partition is stored in the multiple availability zones in the form of multiple copies; the storage layer also maintains the identifier of each physical partition object; The performing data operations on the target data records stored in the multiple copies of the target partition contained in the target physical partition object respectively includes: Based on the maintained identifier of the target physical partition object, searching for multiple replicas of the target partition corresponding to the target physical partition object from multiple replicas of the multiple partitions stored in the multiple availability zones; Data operations are performed on the target data records stored in multiple copies of the target partition respectively.

8. The method according to claim 7, characterized in that The identifier of each physical partition object is an identifier generated based on the schema information corresponding to the physical partition object.

9. The method according to claim 8, characterized in that The identifier of each physical partition object is an identifier generated based on a triple corresponding to the physical partition object, and the triple is a triple generated based on schema information corresponding to the physical partition object; the triple includes an identifier of a data table corresponding to the physical partition object, a primary partition identifier, and a secondary partition identifier.

10. The method according to claim 2, characterized in that The optimizing the execution plan generated based on the data operation request according to the data structure information includes: Determining, according to the schema information, a data encoding rule that matches the data in the target partition targeted by the data operation request; According to the data encoding rule, an execution plan generated based on the data operation request is optimized.

11. The method according to any one of claims 2 to 10, characterized in that: The schema information corresponding to each physical partition object includes at least one of the following information: the name of the data table corresponding to each physical partition object, the index structure of the data table, the field name and field type in the data table, and the partition name and partition level of the partition corresponding to each physical partition object.

12. A data processing device based on a distributed database, characterized in that: The data table in the distributed database is divided into a plurality of partitions; a plurality of physical partition objects corresponding to the plurality of partitions and data structure information corresponding to each physical partition object are maintained in the storage layer of the distributed database; The device comprises: A first acquisition unit, used to acquire a data operation request for a target data table; a determination unit, configured to parse the data operation request to determine a target partition to be operated from a plurality of partitions included in the target data table; A second acquisition unit, configured to acquire the maintained data structure information corresponding to the target physical partition object corresponding to the target partition; An execution unit is used to optimize an execution plan generated based on the data operation request according to the data structure information, and perform data operations on the data stored in the target physical partition object based on the optimized execution plan.

13. A computing device, characterized in that: include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, it performs the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images