Data fragmentation method and device, equipment, medium and program product

Abstract

The invention discloses a data fragmentation method, device and equipment, a medium and a program product, and relates to the technical field of database management. The method comprises the steps of obtaining a current database table structure and a current service query log of each data fragment of the distributed database; according to the current database table structure and the current business query log of each data fragment, constructing a current data distribution relation graph corresponding to each data fragment; identifying the current data distribution relation graph by adopting a data distribution adjustment model, and outputting a target data fragmentation mode of the distributed database; and according to the target data fragmentation mode of the distributed database, performing data distribution adjustment on each data fragment of the distributed database. According to the technical scheme provided by the embodiment of the invention, the query efficiency and load balance of the distributed database are improved.

Description

Technical Field

[0001] This invention relates to the field of database management technology, and in particular to a data sharding method, apparatus, device, medium, and program product. Background Technology

[0002] Distributed databases achieve high availability and scalability by distributing data across multiple data shards, and automatic sharding algorithms are a key technology to ensure efficient data distribution among these nodes. With the advent of the big data era, distributed databases are becoming increasingly common in enterprise applications, cloud services, and financial systems. However, as data volume and query complexity increase, manual sharding becomes inefficient and difficult to adapt to dynamically changing workloads.

[0003] Existing automatic sharding methods for distributed databases typically rely on static rules or simple heuristics. These methods have limitations when dealing with dynamic queries and unevenly distributed data, leading to frequent cross-shard queries and consequently degrading the query performance of the distributed database system. Summary of the Invention

[0004] This invention provides a data sharding method, apparatus, device, medium, and program product, which improves the query efficiency and load balancing of distributed databases.

[0005] According to one aspect of the present invention, a data sharding method is provided, the method comprising:

[0006] Get the current database table structure and current business query logs for each data shard in the distributed database;

[0007] Based on the current database table structure and current business query logs of each data shard, construct the current data distribution relationship diagram corresponding to each data shard;

[0008] A data distribution adjustment model is used to identify the current data distribution relationship diagram and output the target data sharding method of the distributed database.

[0009] Based on the target data sharding method of the distributed database, the data distribution of each data shard in the distributed database is adjusted.

[0010] According to another aspect of the present invention, a data sharding apparatus is provided, the apparatus comprising:

[0011] The current data acquisition module is used to obtain the current database table structure and current business query logs of each data shard in the distributed database.

[0012] The current data distribution graph construction module is used to construct the current data distribution graph corresponding to each data shard based on the current database table structure and current business query log of each data shard.

[0013] The target data sharding output module is used to identify the current data distribution relationship diagram using a data distribution adjustment model and output the target data sharding method of the distributed database.

[0014] The data distribution adjustment module is used to adjust the data distribution of each data shard in the distributed database according to the target data sharding method of the distributed database.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data fragmentation method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data fragmentation method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the data fragmentation method described in any embodiment of the present invention.

[0021] The technical solution of this invention addresses the shortcomings of existing technologies in terms of dynamic adaptability and performance optimization. It can dynamically adjust the data distribution of each data shard in a distributed database based on real-time database table structure and business query logs, exhibiting strong dynamic adaptability. By constructing a current data distribution relationship graph corresponding to each data shard based on the current database table structure and current business query logs, it considers the data association relationships between the data shards in the distributed database. Using a data distribution adjustment model, it identifies the current data distribution relationship graph and outputs the target data sharding method for the distributed database. Based on the target data sharding method, it adjusts the data distribution of each data shard in the distributed database, significantly improving the response speed of the distributed database, reducing the number of cross-shard queries, improving load balancing, and optimizing the query performance of the distributed database. Furthermore, it is suitable for large-scale distributed database systems, especially in scenarios where data volume and query complexity are constantly increasing, exhibiting strong scalability and enhancing the user experience of the distributed database.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a data fragmentation method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a data fragmentation method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is an architectural diagram of a data sharding device applicable to Embodiment 2 of the present invention;

[0027] Figure 4 This is a flowchart of a data fragmentation method provided in Embodiment 2 of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a data sharding device according to Embodiment 3 of the present invention;

[0029] Figure 6This is a schematic diagram of the structure of an electronic device that implements the data fragmentation method of this invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart illustrating a data sharding method provided in Embodiment 1 of the present invention. This embodiment of the invention is applicable to situations involving data sharding adjustments in distributed databases. The method can be executed by a data sharding device, which can be implemented in hardware and / or software. This data sharding device can be configured in an electronic device that carries the data sharding function, such as a client or server.

[0034] See Figure 1 The data sharding methods shown include:

[0035] S101. Obtain the current database table structure and current business query logs for each data shard in the distributed database.

[0036] A distributed database is a database system that stores data across multiple data shards. These data shards are connected via a network to form a logically centralized but physically distributed database.

[0037] The current database table structure represents the data types and corresponding data distribution methods stored in the database tables of a single data shard within the current time period. Optionally, the current database table structure includes fixed and non-fixed table structures. Fixed table structures include column names, column types, column constraints, and column information. Fixed table structures are stored on each data shard. Non-fixed table structures include the data distribution method, such as hash-based distribution, range-based distribution, directory-based distribution, and the corresponding data shard foreign key. The data shard foreign key is the field value upon which the data distribution is based. The data distribution foreign key represents the current data distribution method of the data shard. For example, a data distribution foreign key can be defined using "distribute by (based on...distribution)".

[0038] The current business query log records all business activities within the current time period. For example, the current business query log records all interactions during the process of the application sending a structured query to the database and the database returning the result to the application within the current time period.

[0039] Specifically, by configuring the full database log interface and bypass packet capture functions, the current business query logs of each data shard in the distributed database can be obtained in real time. By parsing the metadata information of the database tables in each data shard of the distributed database, the current database table structure of each data shard can be obtained.

[0040] S102. Based on the current database table structure and current business query logs of each data shard, construct the current data distribution relationship diagram corresponding to each data shard.

[0041] The current data distribution graph is used to represent the data relationships between data shards in a distributed database within the current time period. In the current data distribution graph, nodes represent a specific data shard; edges represent the data relationships between data shards.

[0042] Specifically, each data shard in the distributed database is used as a node in the current data distribution graph. The current database table structure and current business query logs of each data shard are identified, the data relationships between data shards are detected, edges are added between data shards with data relationships, and the current data distribution graph corresponding to each data shard of the distributed database is generated.

[0043] S103. Using a data distribution adjustment model, identify the current data distribution relationship diagram and output the target data sharding method for the distributed database.

[0044] A data distribution adjustment model is used to adjust the data distribution among the data shards of a distributed database. The input data for the data distribution adjustment model is the current data distribution graph; the output is the target data sharding method for each data shard in the distributed database. For example, the data distribution adjustment model is a reinforcement learning model. The target data sharding method is the reference data sharding method among the data shards of the distributed database. The target data sharding method is the adjustment target for the current data sharding method of each data shard in the distributed database.

[0045] Specifically, the current data distribution relationship diagram is input into the data distribution adjustment model, and the target data sharding method between the various data shards of the distributed database is output.

[0046] In an optional embodiment of the present invention, before identifying the current data distribution relationship graph using a data distribution adjustment model and outputting the target data sharding method of the distributed database, the method further includes: obtaining the historical database table structure, historical business query logs, historical shard load, historical query response time, and historical cross-shard query quantity of each data shard in the distributed database; constructing a historical data distribution relationship graph corresponding to each data shard based on the historical database table structure and historical business query logs of each data shard; identifying the historical data distribution relationship graph using an initial model and outputting the estimated data sharding method of the distributed database; calculating the load balancing reward and query efficiency reward corresponding to the estimated data sharding method based on the historical shard load, historical query response time, and historical cross-shard query quantity; and training the initial model to maximize the load balancing reward and query efficiency reward to obtain the data distribution adjustment model.

[0047] Historical database table structures are used to characterize the data types stored in database tables representing individual data shards within a historical time period, as well as the corresponding data distribution methods. Optionally, historical database table structures can include fixed table structures and non-fixed table structures.

[0048] Historical business query logs are used to record all business activities within a historical time period. For example, historical business query logs record all interactions during a historical time period, from when the application sends a structured query to the database to when the database returns the result to the application.

[0049] Historical data distribution graphs are used to represent the data relationships between data shards in a distributed database over a historical period. In a historical data distribution graph, nodes represent a specific data shard, and edges represent the data relationships between data shards.

[0050] The initial model is an untrained data distribution adjustment model. The input data for the initial model is a historical data distribution graph; the output is the estimated data sharding pattern for each data shard in the distributed data. For example, the initial model is a reinforcement learning model. The estimated data sharding pattern is the data sharding pattern among the various data shards of the distributed database within a historical time period predicted by the initial model.

[0051] Historical shard load is used to characterize the load balance of a single data shard in a distributed database. Historical shard load includes historical shard data volume and historical shard workload. Historical shard data volume refers to the amount of data in a single data shard within the distributed database. Historical shard workload is used to assess the activity level of a single data shard in the distributed database. For example, historical shard workload includes CPU (Central Processing Unit) utilization, memory usage, and disk I / O (Input / Output) frequency.

[0052] Historical query response time characterizes the historical execution time of data operations in a distributed database. Historical cross-shard query count represents the number of data shards involved in the data operation within the distributed database. The historical cross-shard query count characterizes whether the data operation can quickly locate the corresponding data shard in the distributed database.

[0053] Load balancing rewards characterize the load balance among data shards in a distributed database when using estimated data sharding. The more balanced the historical load across data shards, the higher the load balancing reward. Query efficiency rewards characterize the query efficiency of data operations in a distributed database when using estimated data sharding. Query efficiency rewards include query response time rewards and cross-shard query quantity rewards. Shorter historical query response times result in higher query efficiency rewards; fewer historical cross-shard queries result in lower query efficiency rewards.

[0054] Specifically, by configuring the full database log interface and bypass packet capture functions, historical business query logs for each data shard of the distributed database are obtained for historical time periods. By parsing the metadata information of the database tables in each data shard of the distributed database for historical time periods, the historical database table structure of each data shard is obtained. Performance monitoring of each data shard of the distributed database is performed to obtain historical shard load, historical query response time, and historical cross-shard query counts for each data shard.

[0055] Specifically, each data shard in the distributed database is used as a node in a historical data distribution graph. The historical database table structure and historical business query logs of each data shard are identified, data relationships between data shards are detected, and edges are added between data shards with data relationships to generate the historical data distribution graph corresponding to each data shard in the distributed database. This historical data distribution graph is then input into the initial model, outputting the estimated data sharding method between the various data shards of the distributed database.

[0056] Specifically, the historical shard load, historical query response time, and historical cross-shard query count are selected from historical shard load, historical query response time, and historical cross-shard query count when using the estimated data sharding method. Based on the historical shard load when using the estimated data sharding method, the load balancing reward corresponding to the estimated data sharding method is calculated. Based on the historical query response time and historical cross-shard query count when using the estimated data sharding method, the query efficiency reward corresponding to the estimated data sharding method is calculated. With the goal of maximizing both the load balancing reward and the query efficiency reward, the initial model is trained to obtain the data distribution adjustment model. There is a pre-defined correlation between historical shard load and load balancing reward. The degree of balance in historical shard load is positively correlated with the load balancing reward. There is a pre-defined correlation between historical query response time, historical cross-shard query count, and query efficiency reward. There is a negative correlation between historical query response time and historical cross-shard query count and query efficiency reward.

[0057] This solution enables offline training of the data distribution adjustment model, improving the load balancing and query efficiency of the data distribution among the data shards in the distributed database determined by the target data sharding method.

[0058] S104. Adjust the data distribution of each data shard in the distributed database according to the target data sharding method of the distributed database.

[0059] Specifically, based on the target data sharding method of the distributed database, the data distribution of each data shard in the distributed database is determined. Based on the data distribution of each data shard in the distributed database, the data distribution of each data shard is adjusted.

[0060] In an optional embodiment of the present invention, the data distribution of each data shard of the distributed database is adjusted according to the target data sharding method of the distributed database, including: determining the target database table structure corresponding to each data shard according to the target data distribution method of the distributed database; creating a new target database table in each data shard according to the target database table structure corresponding to each data shard; and migrating the data in the current database table corresponding to each current database table structure to each target database table.

[0061] The target database table structure is the database table structure corresponding to the target data distribution method in a specific data shard. The target database table structure represents the adjustment target of the current database table structure. Optionally, the target database table structure includes fixed table structures and non-fixed table structures. Fixed table structures include column names, column types, column constraints, and column information. Fixed table structures are stored on each data shard. Non-fixed table structures include the data distribution method, such as hash-based distribution, range-based distribution, directory-based distribution, and corresponding data shard foreign keys. The target database table is the database table generated in the corresponding data shard based on the target database table structure. The current database table is the database table corresponding to the current database table structure. The current database table is the original database table before data migration.

[0062] Specifically, based on the target data distribution method of the distributed database, the target database table structure corresponding to each data shard in the distributed database can be determined. Based on the target database table structure corresponding to each data shard, a new target database table is created in each data shard. Data in the current database tables corresponding to the current database table structure is then migrated to the corresponding target database tables according to the target database table structure.

[0063] This solution improves the efficiency of data distribution adjustment for each data shard in a distributed database by determining the target database table structure for each data shard based on the target data distribution method of the distributed database, creating a new target database table in each data shard according to the target database table structure, and migrating the data in the current database table corresponding to the current database table structure to the target database table.

[0064] In an optional embodiment of the present invention, migrating data from the current database tables corresponding to each current database table structure to each target database table includes: performing a risk assessment of data migration based on each current database table structure and each target database table structure; if the risk assessment is passed, migrating data from the current database tables corresponding to each current database table structure to each target database table; if the risk assessment is not passed, migrating data from the current database tables corresponding to each current database table structure to each target database table during a target idle time period.

[0065] A successful risk assessment means the data migration will have a minimal impact on the business operations corresponding to the distributed database. A failed risk assessment means the data migration will have a significant impact on the business operations corresponding to the distributed database. The target idle time period is the free time during which data migration can be performed. For example, if the business operations corresponding to the distributed database are from 8:00 to 20:00, the target idle time period could be from 20:00 to 8:00 the next day, or sometime between 20:00 and 8:00 the next day. Similarly, if the business operations related to data sharding involved in the data migration are from 9:00 to 12:00, the target idle time period could be from 12:00 to 9:00 the next day, or sometime between 12:00 and 9:00 the next day.

[0066] Specifically, a pre-trained risk assessment model is used to assess the risk of data migration for distributed data based on the differences between the current database table structure and the corresponding target database table structure. If the risk assessment passes, the data in the current database tables corresponding to the current database table structure is directly migrated to the corresponding target database tables according to the target database table structure. If the risk assessment fails, during the target idle time period, the data in the current database tables corresponding to the current database table structure is migrated to the corresponding target database tables according to the target database table structure. The risk assessment model can be a convolutional neural network model. The input data of the risk assessment model is the difference between the current database table structure and the corresponding target database table structure; the output result is whether the risk assessment passes or fails.

[0067] This solution assesses the risk of data migration in a distributed database by analyzing the current and target database table structures. If the risk assessment is successful, the data in the current database tables corresponding to the current database table structures is migrated to the target database tables. If the risk assessment fails, the data in the current database tables corresponding to the current database table structures is migrated to the target database tables during the target idle time period. This avoids the impact of distributed database data migration on business operations and improves the stability of business execution corresponding to the distributed database.

[0068] The technical solution of this invention addresses the shortcomings of existing technologies in terms of dynamic adaptability and performance optimization. It can dynamically adjust the data distribution of each data shard in a distributed database based on real-time database table structure and business query logs, exhibiting strong dynamic adaptability. By constructing a current data distribution relationship graph corresponding to each data shard based on the current database table structure and current business query logs, it considers the data association relationships between the data shards in the distributed database. Using a data distribution adjustment model, it identifies the current data distribution relationship graph and outputs the target data sharding method for the distributed database. Based on the target data sharding method, it adjusts the data distribution of each data shard in the distributed database, significantly improving the response speed of the distributed database, reducing the number of cross-shard queries, improving load balancing, and optimizing the query performance of the distributed database. Furthermore, it is suitable for large-scale distributed database systems, especially in scenarios where data volume and query complexity are constantly increasing, exhibiting strong scalability and enhancing the user experience of the distributed database.

[0069] Example 2

[0070] Figure 2 This is a flowchart of a data sharding method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment of the present invention specifies the step of "constructing a current data distribution relationship graph corresponding to each data shard according to the current database table structure and current business query logs of each data shard" as follows: "Based on the current database table structure of each data shard, detect the common current data distribution methods among the data shards; based on the current business query logs, detect the current common access patterns among the data shards; determine each data shard as the current node in the current data distribution relationship graph; based on the common current data distribution methods and current common access patterns among the data shards, determine the current edge in the current data distribution relationship graph." This approach considers the common data distribution methods and common access patterns among the data shards of the distributed database, improving the efficiency and accuracy of constructing the current data distribution relationship graph of the distributed database. It should be noted that parts not detailed in this embodiment of the present invention can be found in the descriptions of other embodiments.

[0071] See Figure 2 The data sharding methods shown include:

[0072] S201. Obtain the current database table structure and current business query logs for each data shard in the distributed database.

[0073] S202. Based on the current database table structure of each data shard, detect the same current data distribution method among the data shards.

[0074] The current data distribution method refers to the data distribution rules of the database tables corresponding to a certain data shard. For example, data distribution might be based on a hash algorithm using location identifiers or identity identifiers. Database tables with the same data distribution method have the strongest relationships. For instance, Table 1 might be distributed using an identity number based on a hash algorithm, while Table 2 might be distributed using a region code based on a hash algorithm. Due to the different data distribution methods, performing a join operation on Table 1 and Table 2 is more costly, but correspondingly, performing join operations in a distributed database is more efficient. If Table 1 and Table 2 have the same data distribution method, then performing a join operation on Table 1 and Table 2 is less costly, and correspondingly, performing join operations in a distributed database is more efficient.

[0075] Specifically, the process involves examining the foreign keys for data distribution in the current database table structure of each data shard. The foreign keys for data distribution across each data shard are compared. When data shards share the same foreign key for data distribution, it is determined that the corresponding data shards share the same current data distribution pattern. Having the same foreign key for data distribution can be understood as having a foreign key constraint relationship, which is used to characterize the data relationships between tables.

[0076] S203. Based on the current business query logs, detect the current common access patterns among the data shards.

[0077] The current common access mode involves joint operations on database tables corresponding to different data shards. For example, data access modes include insert, delete, update, query, DDL (Data Definition Language) operations, or JOIN operations. Accordingly, the current common access mode can be a JOIN operation.

[0078] Specifically, based on the current business query logs, the joint operations performed on the databases corresponding to different data shards are detected to obtain the current common access patterns among the data shards.

[0079] S204. Determine each data segment as the current node in the current data distribution relationship graph.

[0080] The current node is the node in the current data distribution graph. There is a one-to-one correspondence between the current node and the data shards in the distributed database for the current time period.

[0081] Specifically, each data shard of the distributed database is determined as the current node in the current data distribution graph.

[0082] S205. Based on the common current data distribution method and current common access mode among the data shards, determine the current edge in the current data distribution relationship graph.

[0083] The current edge is an edge in the current data distribution graph. The current edge is used to represent the relationship between different data shards, such as having the same current data distribution method and having a current common access mode.

[0084] Specifically, when data shards have the same current data distribution method or a current common access mode, it is determined that there is a current edge between the current nodes corresponding to the data shards in the current data distribution relationship graph, and a current edge is added between the current nodes corresponding to the data shards in the current data distribution relationship graph.

[0085] In an optional embodiment of the present invention, determining the current edge in the current data distribution graph based on the same current data distribution method and current common access mode among the data shards includes: when the data shards have the same current data distribution method, determining that there is a current edge between the corresponding current nodes of the data shards in the current data distribution graph, and determining that the highest value of the edge weight is the current edge weight of the current edge; when the data shards have a current common access mode, determining the current edge in the current data distribution graph based on the current execution frequency of the current common access mode, and determining the current edge weight of the current edge based on the current execution frequency of the current common access mode.

[0086] The current edge weight is used to characterize the degree of correlation between the data fragments corresponding to the current node connected by the current edge. Based on the current edge weight, the degree of correlation between the data fragments corresponding to different current nodes in the current data distribution graph can be further differentiated. It can be understood that the higher the current edge weight, the higher the correlation between the data fragments corresponding to the current node connected by the current edge; the lower the current edge weight, the lower the correlation between the data fragments corresponding to the current node connected by the current edge.

[0087] The highest value of an edge weight is the highest value within a preset edge weight range. This preset edge weight range is a pre-defined range of edge weights. Optionally, the preset edge weight range can be preset and adjusted by technical personnel. For example, the preset edge weight range can be [0,1]. When data shards share the same current data distribution pattern, the highest value of the edge weight is determined to be the current edge weight. This can be understood as data shards with the same current data distribution pattern having the highest correlation and the lowest cost of performing correlation operations; correspondingly, their edge weights are the highest. The current execution frequency is the execution frequency of the current common access pattern between two data shards in the distributed database within the current time period.

[0088] Specifically, when data shards share the same current data distribution pattern, an edge is determined to exist between the current nodes corresponding to the data shards in the current data distribution graph, and the highest value of the edge weight is determined as the current edge weight. When data shards share a common access pattern, the current execution frequency of the common access pattern is compared with a preset execution frequency threshold. If the current execution frequency is greater than or equal to the preset execution frequency threshold, an edge is determined to exist between the current nodes corresponding to the data shards of the common access pattern in the current data distribution graph, and the current edge weight corresponding to the current execution frequency is determined based on the current execution frequency of the common access pattern. The preset execution efficiency threshold can be a pre-defined minimum execution efficiency value when data shards have a relationship. The preset execution efficiency threshold can also be a pre-defined minimum execution efficiency value when edges are added between nodes corresponding to data shards. Optionally, the preset execution efficiency threshold can be preset and adjusted by technical personnel.

[0089] This scheme takes into account the differences between the same current data distribution method and the current common access mode. Furthermore, when determining whether the current node corresponding to the data shard of the current common access mode in the current data distribution relationship graph has a current edge, it considers the current execution frequency of the current common access mode between the two data shards, which further improves the accuracy of the current data distribution relationship graph.

[0090] S206. Using a data distribution adjustment model, identify the current data distribution relationship diagram and output the target data sharding method for the distributed database.

[0091] S207. Adjust the data distribution of each data shard in the distributed database according to the target data sharding method of the distributed database.

[0092] The technical solution of this invention detects the common current data distribution methods among data shards based on the current database table structure of each data shard, detects the current common access patterns among data shards based on the current business query logs, determines each data shard as the current node in the current data distribution relationship graph, and determines the current edge in the current data distribution relationship graph based on the common current data distribution methods and common access patterns among data shards. This approach considers the common data distribution methods and common access patterns among data shards in a distributed database, thereby improving the efficiency and accuracy of constructing the current data distribution relationship graph of a distributed database.

[0093] Existing automatic sharding methods for distributed databases typically rely on static rules or simple heuristics. These methods have limitations when dealing with dynamic queries and uneven data distribution, leading to frequent cross-shard queries and degraded system query performance. Dynamic queries refer to queries with variable query conditions. Uneven distribution refers to an imbalance in load across data shards. Existing automatic sharding methods for distributed databases perform poorly in dynamic queries.

[0094] For example, existing hash-based data sharding methods cannot distribute data evenly or adapt to changes in data access patterns, leading to some data shards becoming hotspots. This can be understood as hash-based data sharding calculating field values ​​and then distributing data to a fixed data shard. Hash-based data sharding results in high data dispersion. If field values ​​are not highly unique (e.g., 10 field values ​​corresponding to 10 million data points), the data may be concentrated on a few shards, leading to load imbalance. Furthermore, when using hash algorithms for data sharding, such as using location identifiers for data distribution, the load balancing efficiency of location identifiers is poor because some regions have more data than others, resulting in significant differences in data volume between shards. On the other hand, using location identifiers allows for quick location of the specified data shard and retrieval of relevant data during data queries. However, if location identifiers are not used, such as identity identifiers, for data queries without location identifiers, then all data shards will be queried, making it impossible to quickly locate the corresponding data shard and retrieve the corresponding data quickly, resulting in low query efficiency.

[0095] For example, while range-based and directory-based data sharding methods can allocate data shards according to data ranges, they can easily lead to uneven data distribution, especially when data grows or data access patterns change. Range-based data sharding can be understood as using a unique identifier, with all data increasing sequentially based on this identifier. Different data ranges correspond to different data shards. For example, 1-10000 is in data shard 1; 10001-20000 in data shard 2; and 20001-30000 in data shard 3. With a certain amount of existing data, the data distribution across shards is relatively balanced, but this scenario has significant limitations. For instance, in other scenarios, if range-based data sharding is used, such as sharding by month, due to business periods of low and high activity (where data volume is small during low periods and large during peak periods), uneven data distribution across shards can occur.

[0096] Based on the above embodiments, Figure 3 This is a preferred embodiment of a data sharding device provided by the present invention. For example... Figure 3 As shown, the present invention includes a monitoring module 301, a graph representation module 302, a reinforcement learning agent 303, and a sharding adjustment module 304.

[0097] The monitoring module (i.e., the current data acquisition module) 301 is deployed on each data shard of the distributed database, collecting the current database table structure, current business query logs, and current performance data in real time. The current performance data includes the current shard load, current query response time, and current number of cross-shard queries. Regarding current performance data collection, for the current query response time, the time interval from the user initiating a query request to the database returning the query result is accurately recorded through bypass packet capture to obtain the current query response time, which is used to measure the speed at which the distributed database processes query tasks. For the current shard load, the workload carried by each data shard is monitored in real time, including but not limited to key indicators such as CPU utilization, memory usage, and disk I / O frequency, to assess the busyness of each data shard. For the current number of cross-shard queries, the number of query operations involving multiple data shards or the number of data shards involved in a single query operation is counted within a certain time period.

[0098] The graph representation module (i.e., the current data distribution graph construction module) 302 presents the complex relationships between database table structures and data shards in an intuitive and easy-to-understand graph structure. Each node represents a data shard; each edge represents the data association between data shards, such as inherent relationships between data (i.e., the same current data distribution method) or common access patterns. For relationships established between database tables via shard keys, this is represented in the graph structure as edges between the corresponding nodes. For example, if the "Order Table" and the "Customer Table" are associated via customer identifiers, then in the graph representation (current data distribution graph), there will be an edge between the node representing the "Order Table shard" and the node representing the "Customer Table shard," thus visually demonstrating their data association. Table relationships involved in frequent JOIN operations are also represented in the graph structure. If multiple tables frequently use JOIN operations to jointly query data, the nodes of the corresponding data shards will be interconnected by edges. Edge weights can be set based on the current execution frequency of the JOIN operation; the higher the execution frequency, the greater the edge weight, thus more clearly reflecting the close shared access patterns between the data. A graph database can be used to construct and store this graph structure (i.e., the current data distribution graph), providing a solid data foundation for subsequent interactions between reinforcement learning agents.

[0099] The reinforcement learning agent (i.e., the target data sharding output module) 303 uses deep reinforcement learning algorithms (such as Deep Q-Network (DQN) or Actor-Critic algorithm) to learn sharding strategies. The reinforcement learning agent can model the sharding management problem as a Markov Decision Process (MDP). In the reinforcement learning agent, the state is defined as the current data distribution (i.e., the current data distribution graph). The reward function is determined based on two core objectives: reducing the number of cross-shard queries and balancing the shard load. A higher reward value is obtained when the reinforcement learning agent's estimated data sharding method effectively reduces the number of cross-shard queries or makes the load of each data shard more balanced; conversely, a lower reward value is obtained if the action (i.e., the estimated data sharding method adopted by the reinforcement learning agent) leads to an increase in the number of cross-shard queries or exacerbates the imbalance of shard load. By continuously trying different predicted data partitioning methods and adjusting them based on reward feedback, reinforcement learning agents can gradually learn the optimal action sequence under various states, i.e., the optimal partitioning strategy (i.e., the optimal data partitioning method). In terms of algorithm implementation, Deep Q-Networks (DQNs) and their improved algorithms, such as DoubleDQN or Dueling DQN, can be used. These algorithms leverage the powerful function fitting capabilities of neural networks to efficiently learn the complex mapping relationship between states and actions, thereby quickly finding a target data partitioning method that approximates the optimal partitioning strategy.

[0100] The sharding adjustment module (i.e., the data distribution adjustment module) 304 performs data migration, shard splitting, or merging operations based on the reinforcement learning agent's decision (i.e., the target data sharding method). The sharding adjustment module is the execution unit in the system responsible for putting the decision suggestions generated by the reinforcement learning agent into practice. Upon receiving the sharding adjustment suggestion (i.e., the target data sharding method) from the reinforcement learning agent, it executes the corresponding operation, calling the distributed database redistribution interface to ensure that data can be accurately moved from one shard to another. Furthermore, to reduce computational overhead, the reinforcement learning agent (i.e., the data distribution adjustment model) can be trained offline periodically and applied online to adapt to new data or changes in data access patterns.

[0101] The data sharding device based on a distributed database provided by this invention is based on the use of reinforcement learning to dynamically manage the various data shards of the distributed database, so as to minimize cross-shard queries and achieve load balancing.

[0102] Based on the above embodiments, Figure 4This is a preferred embodiment of a data fragmentation method provided by the present invention. See also Figure 4 The data sharding methods shown include:

[0103] S401, Data Collection.

[0104] Specifically, the monitoring module continuously collects performance data (including shard load, query response time, and cross-shard query count) from the monitored servers (i.e., each data shard). It obtains business query logs in real time by configuring a full database log interface and bypass packet capture. By parsing the metadata information of the database tables in each data shard of the distributed database, it obtains the database table structure of each data shard.

[0105] S402, Graph Construction.

[0106] The graph representation module constructs or updates the graph structure (i.e., the data distribution graph) based on the performance data and business query logs collected by the monitoring module. In constructing the data distribution graph, the database table structure and data relationships of each data shard in the distributed database are first analyzed.

[0107] (1) By parsing the metadata information of the database tables in each data shard of the distributed database, the foreign key constraint relationship between the database tables is identified, and edges are added between the nodes of the data shards corresponding to the database tables with foreign key constraint relationships.

[0108] (2) Identify frequent JOIN operations. The graph representation module will perform in-depth analysis of the business query logs to count the execution frequency and association conditions of JOIN operations between database tables in different data shards. If the number of JOIN operations between two database tables in a certain time period (i.e., the current time period or the historical time period) exceeds the set threshold (i.e., the preset execution frequency threshold), an edge will be added between the nodes corresponding to the data shards, and the edge will be assigned a corresponding weight (i.e., edge weight) according to the JOIN operation frequency (i.e., the current execution frequency or the historical execution frequency).

[0109] Optionally, when the data distribution of each data shard in the distributed database changes or the data access pattern changes, the graph representation module will trigger the graph structure (i.e., the data shard relationship graph) update mechanism.

[0110] This includes changes in the data distribution of the distributed database, such as an increase in the data volume of data shards or the addition of new tables in the distributed database, including changes in the data volume of existing tables in data shards or the addition of new tables (i.e., the emergence of new business logic). For example, initially, the business logic involved transactions between table a corresponding to data shard 1 and table b corresponding to data shard 2. However, due to changes in the business logic, the transactions now involve transactions between table a in data shard 1 and table c in data shard 3, and the join conditions between tables a and b are different from those between tables a and c. Changes in data access patterns, such as the emergence of new complex query operations, also contribute to this change.

[0111] At this point, the business query logs and metadata information of each data shard in the distributed database can be rescanned to obtain the current database table structure and current business query logs. Based on the current database table structure and current business query logs, the changed parts are detected. For newly added database tables or data relationships, corresponding nodes and edges are added to the current data distribution graph; for database tables or data relationships that are no longer used, the corresponding nodes and edges are deleted.

[0112] S403, Strategy Learning.

[0113] The reinforcement learning agent takes the data distribution graph generated by the graph representation module as input and learns the optimal partitioning policy. In practice, the reinforcement learning agent explores policies in a simulated environment. The agent first encodes the data distribution graph into a vector form suitable for neural network processing and inputs it into the deep learning neural network. Based on the state vector of the input data distribution graph, the deep learning neural network outputs the estimated value of each action (i.e., the predicted data partitioning method).

[0114] The reinforcement learning agent simulates the execution of a shard key change strategy (i.e., predicting data sharding methods). During the simulation, based on the data distribution graph and performance data, it calculates the potential performance impact of this action. For example, it calculates the changes in the number of cross-shard queries and the changes in the load on each shard after the shard movement. Based on these performance changes, a reward value is calculated according to a pre-defined reward function. If changing the shard key significantly reduces cross-shard queries and makes the shard load more balanced, a higher reward is given; conversely, if it leads to a decrease in query performance or an exacerbation of load imbalance, a lower reward is given.

[0115] The agent accumulates a large amount of state-action-reward data in a simulated environment by repeatedly performing the above process. Using this data, the neural network is trained using an experience replay mechanism and gradient descent algorithm. The network parameters are adjusted so that the network can more accurately predict the value of different actions, thereby learning the optimal partitioning strategy.

[0116] S404, Segmentation Adjustment.

[0117] In practical use, after receiving the reinforcement learning agent's suggestion (i.e., the target data distribution method), the sharding adjustment module begins to execute specific data migration operations. Before data migration, a risk assessment and planning can be performed on the migration task. The data migration plan is determined based on the amount of data to be migrated and the load conditions of the source and target nodes.

[0118] Optionally, after the sharding adjustment operation is completed, the system enters a feedback loop phase. The monitoring module monitors the performance data of the distributed database in real time after the adjustment and compares it with the performance data of the distributed database before the adjustment. The effectiveness of the sharding adjustment strategy is evaluated by calculating key indicators such as the rate of change in query response time, the reduction in the number of cross-shard queries, and the improvement in shard load balancing. This performance data is then transmitted again to the reinforcement learning agent, which uses it as new training data to update the data distribution adjustment model. During the update process, the reinforcement learning agent adjusts the parameters of the data distribution adjustment model based on the new data, optimizing the reward function so that the data distribution adjustment model can better adapt to the new data distribution and data access patterns. The entire feedback loop repeats continuously, and over time, the data distribution adjustment model continuously optimizes the sharding strategy, ensuring that the distributed database always maintains a high-efficiency operating state.

[0119] This invention is a data sharding method based on distributed databases. It uses a reinforcement learning model to automatically learn the optimal sharding strategy, solving the problem that existing technologies rely heavily on static rules and struggle to adapt to dynamic data access patterns. By modeling the database table structure and data access patterns as a graph structure, and using graph representation to model data relationships, it addresses the lack of systematic analysis of data associations in existing technologies. Through real-time monitoring of performance data and data access patterns, it automatically adjusts sharding to reduce cross-shard queries, avoiding manual intervention and achieving real-time monitoring of performance data and data access patterns as well as automatic adjustment of data sharding. Load balancing is achieved through shard adjustment, reducing hotspots in some data shards and solving the problem of low query efficiency when data distribution is uneven, thus achieving load balancing optimization. These improvements address the shortcomings of existing technologies in dynamic adaptability and performance optimization, enhancing the scalability and user experience of distributed databases, and are particularly suitable for high-concurrency scenarios. This invention can dynamically adjust the sharding strategy based on real-time performance data and data access patterns, exhibiting strong dynamic adaptability. By reducing cross-shard queries and balancing the load, it significantly improves database response speed and optimizes the query performance of distributed databases. It is suitable for large-scale distributed database systems, especially in scenarios where data volume and query complexity are constantly increasing, and has strong scalability. It solves the shortcomings of existing technologies in terms of dynamic adaptability and performance optimization, improves database scalability and user experience, and is particularly suitable for high-concurrency and high-load scenarios.

[0120] Example 3

[0121] Figure 5 This is a schematic diagram of a data sharding device provided in Embodiment 3 of the present invention. This embodiment of the invention is applicable to situations involving data sharding adjustments in distributed databases. The device can execute data sharding methods and can be implemented in hardware and / or software. It can be configured in an electronic device that carries data sharding functionality, such as a client or server.

[0122] See Figure 5 The data sharding device shown includes: a current data acquisition module 501, a current data distribution graph construction module 502, a target data sharding mode output module 503, and a data distribution adjustment module 504. Specifically, the current data acquisition module 501 acquires the current database table structure and current business query logs of each data shard in the distributed database; the current data distribution graph construction module 502 constructs a current data distribution graph corresponding to each data shard based on the current database table structure and current business query logs; the target data sharding mode output module 503 identifies the current data distribution graph using a data distribution adjustment model and outputs the target data sharding mode of the distributed database; and the data distribution adjustment module 504 adjusts the data distribution of each data shard in the distributed database according to the target data sharding mode.

[0123] The technical solution of this invention addresses the shortcomings of existing technologies in terms of dynamic adaptability and performance optimization. It can dynamically adjust the data distribution of each data shard in a distributed database based on real-time database table structure and business query logs, exhibiting strong dynamic adaptability. By constructing a current data distribution relationship graph corresponding to each data shard based on the current database table structure and current business query logs, it considers the data association relationships between the data shards in the distributed database. Using a data distribution adjustment model, it identifies the current data distribution relationship graph and outputs the target data sharding method for the distributed database. Based on the target data sharding method, it adjusts the data distribution of each data shard in the distributed database, significantly improving the response speed of the distributed database, reducing the number of cross-shard queries, improving load balancing, and optimizing the query performance of the distributed database. Furthermore, it is suitable for large-scale distributed database systems, especially in scenarios where data volume and query complexity are constantly increasing, exhibiting strong scalability and enhancing the user experience of the distributed database.

[0124] In an optional embodiment of the present invention, the current data distribution graph construction module 502 includes: a current data distribution mode detection unit, configured to detect the same current data distribution mode among the data shards based on the current database table structure of each data shard; a current common access mode detection unit, configured to detect the current common access mode among the data shards based on the current business query log; a current node determination unit, configured to determine each data shard as the current node in the current data distribution graph; and a current edge determination unit, configured to determine the current edge in the current data distribution graph based on the same current data distribution mode and the current common access mode among the data shards.

[0125] In an optional embodiment of the present invention, the current edge determination unit includes: a first current edge determination subunit, configured to determine, when the data shards have the same current data distribution mode, that there is a current edge between the corresponding current nodes of the data shards in the current data distribution relationship graph, and determine that the highest value of the edge weight is the current edge weight of the current edge; and a second current edge determination subunit, configured to determine the current edge in the current data distribution relationship graph according to the current execution frequency of the current common access mode when the data shards have the current common access mode, and determine the current edge weight of the current edge according to the current execution frequency of the current common access mode.

[0126] In an optional embodiment of the present invention, the apparatus further includes: a historical data acquisition module, configured to acquire, before identifying the current data distribution relationship graph using the data distribution adjustment model and outputting the target data sharding method of the distributed database, the historical database table structure, historical business query logs, historical shard load, historical query response time, and historical cross-shard query quantity of each data shard in the distributed database; a historical data distribution relationship graph construction module, configured to construct a historical data distribution relationship graph corresponding to each data shard based on the historical database table structure and historical business query logs of each data shard; a predicted data sharding method output module, configured to identify the historical data distribution relationship graph using the initial model and output the predicted data sharding method of the distributed database; a predicted data sharding mode reward calculation module, configured to calculate the load balancing reward and query efficiency reward corresponding to the predicted data sharding method based on the historical shard load, the historical query response time, and the historical cross-shard query quantity; and a data distribution adjustment model training module, configured to train the initial model to maximize the load balancing reward and the query efficiency reward to obtain the data distribution adjustment model.

[0127] In an optional embodiment of the present invention, the data distribution adjustment module 504 includes: a target database table structure determination unit, configured to determine the target database table structure corresponding to each data shard according to the target data distribution method of the distributed database; a target database table creation unit, configured to create a target database table in each data shard according to the target database table structure corresponding to each data shard; and a data migration unit, configured to migrate the data in the current database table corresponding to each current database table structure to each target database table.

[0128] In an optional embodiment of the present invention, the data migration unit includes: a risk assessment subunit, configured to perform a risk assessment of data migration based on each of the current database table structures and each of the target database table structures; a first data migration unit, configured to migrate data from the current database table corresponding to each of the current database table structures to each of the target database tables when the risk assessment passes; and a second data migration unit, configured to migrate data from the current database table corresponding to each of the current database table structures to each of the target database tables during a target idle time period when the risk assessment fails.

[0129] The data sharding apparatus provided in the embodiments of the present invention can execute the data sharding method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0130] In the technical solutions of this invention, the acquisition, storage, and application of the current database table structure, current business query log, historical database table structure, historical business query log, historical shard load, historical query response time, and historical cross-shard query quantity of each data shard in the distributed database all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0131] Example 4

[0132] Figure 6 A schematic diagram of an electronic device 600 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0133] like Figure 6As shown, the electronic device 600 includes at least one processor 601 and a memory, such as a read-only memory (ROM) 602 or a random access memory (RAM) 603, communicatively connected to the at least one processor 601. The memory stores computer programs executable by the at least one processor. The processor 601 can perform various appropriate actions and processes based on the computer program stored in the ROM 602 or loaded into the RAM 603 from storage unit 608. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0134] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 601 performs the various methods and processes described above, such as data sharding methods.

[0136] In some embodiments, the data fragmentation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by processor 601, one or more steps of the data fragmentation method described above may be performed. Alternatively, in other embodiments, processor 601 may be configured to perform the data fragmentation method by any other suitable means (e.g., by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0142] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0143] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data sharding method, characterized in that, The method includes: Get the current database table structure and current business query logs for each data shard in the distributed database; Based on the current database table structure and current business query logs of each data shard, construct the current data distribution relationship diagram corresponding to each data shard; A data distribution adjustment model is used to identify the current data distribution relationship diagram and output the target data sharding method of the distributed database. Based on the target data sharding method of the distributed database, the data distribution of each data shard in the distributed database is adjusted.

2. The method according to claim 1, characterized in that, The step of constructing a current data distribution relationship diagram corresponding to each data shard based on the current database table structure and current business query log of each data shard includes: Based on the current database table structure of each data shard, detect the common current data distribution methods among each data shard; Based on the current business query logs, detect the current common access patterns among the data shards; Each of the aforementioned data fragments is determined as the current node in the current data distribution relationship graph; The current edge in the current data distribution graph is determined based on the common current data distribution method and the current common access mode among the data shards.

3. The method according to claim 2, characterized in that, The step of determining the current edge in the current data distribution graph based on the common current data distribution method and the current common access mode among the data fragments includes: When the data shards have the same current data distribution method, it is determined that there is a current edge between the current nodes corresponding to the data shards in the current data distribution relationship graph, and the highest value of the edge weight is determined to be the current edge weight of the current edge. When there is a current common access mode between the data shards, the current edge in the current data distribution graph is determined according to the current execution frequency of the current common access mode, and the current edge weight of the current edge is determined according to the current execution frequency of the current common access mode.

4. The method according to claim 1, characterized in that, Before adopting the data distribution adjustment model to identify the current data distribution relationship diagram and output the target data sharding method of the distributed database, the method further includes: Obtain the historical database table structure, historical business query logs, historical shard load, historical query response time, and historical cross-shard query count for each data shard in the distributed database. Based on the historical database table structure and historical business query logs of each data shard, construct a historical data distribution relationship diagram corresponding to each data shard; Using an initial model, the historical data distribution relationship diagram is identified, and the estimated data sharding method of the distributed database is output. Based on the historical shard load, the historical query response time, and the historical cross-shard query count, calculate the load balancing reward and query efficiency reward corresponding to the estimated data sharding method; The initial model is trained by maximizing the load balancing reward and the query efficiency reward to obtain the data distribution adjustment model.

5. The method according to claim 1, characterized in that, The step of adjusting the data distribution of each data shard in the distributed database according to the target data sharding method of the distributed database includes: Based on the target data distribution method of the distributed database, determine the target database table structure corresponding to each data shard; Based on the target database table structure corresponding to each data shard, a new target database table is created in each data shard; Migrate the data in the current database table corresponding to each of the current database table structures to each of the target database tables.

6. The method according to claim 5, characterized in that, The step of migrating data from the current database table corresponding to each of the current database table structures to each of the target database tables includes: Based on the current database table structure and the target database table structure, a risk assessment for data migration is conducted. When the risk assessment is passed, the data in the current database table corresponding to each of the current database table structures will be migrated to each of the target database tables. If the risk assessment fails, during the target idle time period, the data in the current database table corresponding to each of the current database table structures will be migrated to each of the target database tables.

7. A data sharding device, characterized in that, The device includes: The current data acquisition module is used to obtain the current database table structure and current business query logs of each data shard in the distributed database. The current data distribution graph construction module is used to construct the current data distribution graph corresponding to each data shard based on the current database table structure and current business query log of each data shard. The target data sharding output module is used to identify the current data distribution relationship diagram using a data distribution adjustment model and output the target data sharding method of the distributed database. The data distribution adjustment module is used to adjust the data distribution of each data shard in the distributed database according to the target data sharding method of the distributed database.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data fragmentation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data fragmentation method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data fragmentation method according to any one of claims 1-6.

Images