Data fast sharding method in distributed database environment and electronic device

Abstract

This application provides a method for fast data sharding in a distributed database environment and an electronic device. The method includes: acquiring node lifecycle information, node load information, and data access pattern information of multiple nodes; determining a dependency graph between the multiple shards based on the data access pattern information; determining a sharding strategy based on the node lifecycle information, node load information, data access pattern information, and the dependency graph; and performing shard splitting or shard merging operations on the multiple shards based on the dependency graph and the sharding strategy. This scheme constructs a dependency graph between shards and quantifies dependency strength, accurately identifying the dependencies of related shards before performing shard splitting or merging operations, avoiding access path interruptions or dependency disruptions caused by splitting, and enhancing the modularity and maintainability of the system.

Description

Technical Field

[0001] This application relates to the field of data sharding, and more specifically, to a method for fast data sharding in a distributed database environment and an electronic device. Background Technology

[0002] With the rapid development of big data technology, cloud computing, and internet services, traditional single-machine databases have gradually revealed bottlenecks in terms of data scale, processing efficiency, and access concurrency capabilities. Consequently, to meet the high availability, high performance, and high scalability requirements of modern application systems, an increasing number of database systems have adopted distributed database architectures. This architecture divides data into multiple shards and distributes them across different physical nodes, achieving horizontal scaling and load balancing, thereby effectively supporting large-scale, high-concurrency business needs.

[0003] However, most existing sharding strategies are based on static planning in the early stages of deployment, and when performing operations such as shard splitting, merging, and migration, they often cause problems such as access conflicts, resource contention, or service blocking, which seriously limit the maintainability and flexibility of the system. Summary of the Invention

[0004] This application addresses some shortcomings mentioned in the background technology by providing a method and electronic device for rapid data sharding in a distributed database environment. By constructing a dependency graph between shards and quantifying dependency strength, this application achieves accurate identification and dynamic management of complex dependencies between shards, fully considering the impact of dependencies when performing shard splitting or merging operations. Furthermore, it avoids problems such as access conflicts, resource contention, or service blocking caused by undecoupled dependencies, significantly improving the maintainability and flexibility of the distributed database system.

[0005] Firstly, a method for fast data sharding in a distributed database environment is provided. This method includes: obtaining node lifecycle information, node load information, and data access pattern information of multiple nodes, wherein the node lifecycle information is used to predict the node's offline time; determining a dependency graph between the multiple shards based on the data access pattern information, wherein the dependency graph includes dependent nodes and dependent edges, the dependent nodes indicating the shards, the dependent edges indicating the data access dependencies between the shards, and each dependent edge also including a dependency weight indicating the strength of the dependency relationship between two shards; determining a sharding strategy based on the node lifecycle information, the node load information, the data access pattern information, and the dependency graph, the sharding strategy indicating which shards among the multiple shards should be split or merged; and performing shard splitting or shard merging operations on the multiple shards based on the dependency graph and the sharding strategy.

[0006] This scheme constructs a dependency graph between shards and quantifies dependency strength. Before performing shard splitting or merging operations, it accurately identifies the dependencies between related shards, preventing access path interruptions or dependency disruptions caused by splitting, thus enhancing the system's modularity and maintainability. Simultaneously, it dynamically formulates sharding strategies based on node load information and data access pattern information, resulting in a more reasonable number and size of shards, reducing frequent re-sharding operations, and improving overall system performance.

[0007] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: prioritizing the fragments to be split or merged as indicated in the fragmentation strategy according to the dependency strength in the dependency graph, wherein fragments with higher dependency strength have higher priority than fragments with lower dependency strength; performing fragment splitting or merging operations on the multiple fragments according to the fragmentation strategy includes: performing fragment splitting or merging operations according to the fragmentation strategy and the result of the priority ranking.

[0008] This scheme prioritizes the processing of shards with strong dependencies, ensuring that highly coupled shards are operated on first during adjustments, effectively reducing the risk of data access anomalies or performance bottlenecks caused by shard adjustments.

[0009] In conjunction with the first aspect, in a possible implementation of the first aspect, the strength of the dependency is determined based on the frequency of access to the same data item by the two shards and the number of dependency connections between the two shards.

[0010] This scheme uses data from two dimensions—access frequency and the number of dependent connections—to quantify dependency strength, making dependency assessment more accurate and objective.

[0011] In conjunction with the first aspect, in a possible implementation of the first aspect, the dependencies between the shards include circular dependencies, which are used to indicate that the dependencies between multiple shards form a closed loop. Before performing the shard splitting or shard merging operation indicated in the sharding strategy on the multiple shards, the method further includes: if the circular dependency is detected, performing at least one of the following processing strategies: abstracting a public interface or service to transform the circular dependency into a unidirectional dependency; using asynchronous messages or an event bus to replace direct calls; or using a target mechanism in the framework to resolve the startup deadlock problem.

[0012] This solution can accurately identify circular dependencies between shards and formulate differentiated processing strategies, turning bidirectional call relationships into unidirectional call relationships, breaking circular dependencies, and significantly reducing the coupling between services.

[0013] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: determining directed edges to obtain a directed graph based on the dependency direction of the partition in the dependency graph; starting from each partition, detecting cycles in the directed graph and marking node states, including unvisited, being visited, and visited; if a node being visited is detected, determining that a directed cycle exists in the directed graph and determining that the circular dependency exists.

[0014] This scheme enables efficient and accurate identification of circular dependencies between parts of a directed graph through loop detection.

[0015] In conjunction with the first aspect, in a possible implementation of the first aspect, the dependency relationship between the shards includes cross dependency, which is used to indicate that two or more shards depend on the same set of data. Before performing the shard splitting or shard merging operation indicated in the sharding strategy on the multiple shards, the method further includes: if the cross dependency is detected, identifying the hot shard in the cross dependency; performing at least one of the following processing strategies on the hot shard: performing cache optimization for the hot shard; and allocating the data of the hot shard to local maintenance within the module.

[0016] This solution can accurately identify cross-dependency hotspots between shards and take targeted governance measures. By introducing a caching mechanism and module-level local data maintenance, it significantly reduces access latency and resource contention caused by cross-dependencies and reduces the resource conflict rate between shards.

[0017] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: detecting the frequency of inter-service calls and resource sharing; if the resource sharing is greater than a preset sharing threshold, determining that there is a cross dependency between the shards; and determining the shard frequently called by multiple upstream sources as the hot shard based on the frequency of inter-service calls.

[0018] This solution utilizes the frequency of calls and resource sharing between monitoring services to achieve automated identification of cross dependencies and hotspots.

[0019] In conjunction with the first aspect, in a possible implementation of the first aspect, obtaining the node load information and data access pattern information of multiple nodes includes: obtaining the real-time load status and data access pattern of the multiple nodes in the distributed database; and determining the node load information and data access pattern information of the multiple nodes within a future preset time period based on the real-time load status and data access pattern of the multiple nodes and a pre-trained prediction model.

[0020] This solution utilizes predictive models to anticipate node load trends and data access patterns over future time periods, enabling sharding strategies to respond in advance to business peaks or resource pressure changes, thus avoiding performance bottlenecks and system instability caused by sudden traffic surges or concentrated access.

[0021] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the sharding strategy based on the node load information and the data access mode information includes: determining the current system load distribution status, node lifecycle changes, and sharding resource requirements based on the node load information and the data access mode information; and determining the sharding strategy based on the load distribution status, node lifecycle changes, and resource requirements.

[0022] This approach comprehensively considers multiple factors such as system load distribution, node lifecycle changes, and sharding resource requirements to formulate sharding strategies, making sharding adjustments more closely aligned with the actual operating status of the system.

[0023] In conjunction with the first aspect, in a possible implementation of the first aspect, the method further includes: determining the dependency mismatch risk between the shards based on the dependency weight indicated in the dependency graph; and, if the dependency mismatch risk is greater than a preset risk threshold, performing at least one of the following adjustment measures: adjusting the load of the shards; transferring part of the load to other nodes; optimizing the access path of the shards; migrating the shards to a more suitable node; or merging or splitting two shards to reduce cross-shard dependencies.

[0024] This solution uses the linkage between dependency strength and node load to determine the risk of dependency mismatch between shards. It can provide timely warnings and automatically adjust the shard running status when the dependency path and node load status do not match, effectively preventing performance overload and service congestion caused by broken dependency chains or access path failures.

[0025] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the dependency mismatch risk between the shards based on the dependency weight indicated in the dependency graph includes: determining the dependency mismatch risk between the two shards based on the dependency strength between the two shards, the current load of the two shards, the communication delay between the two shards, and a preset adjustment coefficient, wherein the adjustment coefficient is used to adjust the degree of influence of load difference on the dependency mismatch risk.

[0026] This scheme incorporates dependency strength, node load, and communication latency into a unified risk assessment formula, enabling a quantitative assessment of dependency mismatch risk.

[0027] In a second aspect, an electronic device is provided, comprising one or more processors and one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the method described in the first aspect or any possible implementation thereof to be performed.

[0028] Thirdly, a computer-readable storage medium is provided that stores computer instructions that, when executed on a computer, cause the method described in the first aspect or any possible implementation thereof to be performed.

[0029] Fourthly, a computer program product is provided that, when run on a computer, causes the computer to perform the method as described in the first aspect or any possible implementation thereof. Attached Figure Description

[0030] Figure 1 A schematic flowchart of a fast data sharding method in a distributed database environment provided in an embodiment of this application is shown; Figure 2 A schematic diagram illustrating the steps of a fast data fragmentation method provided in an embodiment of this application is shown; Figure 3 A schematic diagram illustrating the steps of the fragmentation process based on inter-fragment dependencies is shown. Figure 4 This is a structural schematic diagram of a device provided in an embodiment of this application; Figure 5 This is a structural schematic diagram of a system on a chip (SoC) provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0032] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0033] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] As mentioned earlier, most existing sharding strategies are based on static planning in the early stages of deployment. When performing operations such as shard splitting, merging, and migration, problems such as access conflicts, resource contention, or service blocking often arise, which seriously limits the maintainability and flexibility of the system.

[0035] Specifically, in actual operation, shards exhibit complex logical dependencies, such as circular dependencies (forming dependency loops) or cross dependencies (multiple shards jointly depend on the same data resource). Existing sharding strategies typically fail to provide accurate identification and dynamic analysis of these dependency structures. This leads to access conflicts, resource contention, or service blocking issues during shard splitting, merging, and migration operations due to undecoupled dependencies, severely limiting system maintainability and flexibility. Furthermore, once the dependencies between shards deviate from the actual node load status—for example, a highly dependent shard hosted on a low-performance or soon-to-fail node—it can easily cause dependency chain breaks, access path failures, or performance overload. Existing sharding strategies lack a mechanism for judging based on the linkage between dependency strength and node load, failing to identify such dependency mismatch risks in advance and lacking corresponding adaptive adjustment strategies, resulting in hidden failure risks during system operation.

[0036] Furthermore, most existing sharding strategies are based on static planning during the initial deployment phase, failing to automatically optimize the sharding structure during operation based on real-time changes in access frequency, hotspot migration, and data hot / cold evolution. This static strategy easily leads to some shards remaining under high load for extended periods, while other shards waste resources, resulting in severe load imbalance and impacting the overall system response performance and stability. Additionally, current systems lack lifecycle monitoring mechanisms for node online / offline or fault states, making it impossible to identify the risk of nodes going offline or exhausting resources in advance. When a node abnormally exits, service interruptions or data unavailability often occur due to untimely shard migration, affecting business continuity and increasing system recovery costs. Currently, most sharding adjustments rely on historical average loads or manual configuration, lacking the ability to model and predict node load trends and data access density evolution over future periods, making it difficult to respond promptly to business peaks or resource pressure changes. This passive adjustment strategy cannot avoid performance bottlenecks and system instability caused by sudden traffic surges or concentrated access.

[0037] To address the aforementioned issues, this application proposes a fast data sharding method 100 in a distributed database environment. Figure 1 A schematic flowchart of a fast data sharding method 100 in a distributed database environment provided in an embodiment of this application is shown. Figure 1 As shown, method 100 includes steps S110 to S140. In method 100, by constructing a dependency graph between shards and quantifying the dependency strength, the method achieves accurate identification and dynamic governance of complex dependencies between shards. When performing shard splitting or merging operations, the method can fully consider the impact of dependencies, avoid access conflicts, resource contention, or service blocking caused by undecoupled dependencies, and significantly improve the maintainability and flexibility of the distributed database system.

[0038] Step S110: Obtain node lifecycle information, node load information, and data access mode information for multiple nodes.

[0039] The node's lifecycle information is used to predict its offline time.

[0040] In the embodiments of this application, a node can refer to a physical server or virtual machine instance in a distributed database cluster, with each node responsible for storing and processing one or more data shards. Sharding involves dividing a logically complete database into several parts according to certain rules; each part is called a shard and is distributed across different physical nodes. In the embodiments of this application, the node load information includes metrics such as CPU utilization, memory utilization, and network bandwidth usage, while the data access pattern information includes access frequency, concurrency, and access to hot data.

[0041] For example, embodiments of this application can obtain the lifecycle information of each node through the monitoring and logging mechanism in a distributed database sharding system. This lifecycle information includes the node's status, data load and system load information when it goes online and offline, and the status includes online, offline, load, and resource consumption. Furthermore, the start and end of each node's lifecycle can be marked using timestamps and status markers, and the node's resource consumption, load data, network bandwidth, and memory usage can be recorded.

[0042] As another example, embodiments of this application can use log analysis tools to record key events in the node lifecycle in real time, and perform statistical analysis on the node's workload, resource consumption, and data access status during the lifecycle to obtain the node's resource consumption status and data access patterns at different lifecycle stages. These key events include going online, going offline, and status changes. Simultaneously, system monitoring tools acquire the real-time load status and data access patterns of each node and store them in a data sharding monitoring database. Furthermore, through this data sharding monitoring database, historical data on the node's real-time load status and data access patterns are obtained, and a trained prediction model is used to predict the node load and data access patterns for future time periods.

[0043] Optionally, in embodiments of this application, a Long Short-Term Memory network and an autoregressive integrated moving average (ARIMA) algorithm can be used to predict node load and data access patterns. Specifically, firstly, the real-time load status and data access patterns of the multiple nodes in the distributed database are obtained. Then, based on the real-time load status and data access patterns of the multiple nodes and the pre-trained ARIMA prediction model, the node load information and data access pattern information of the multiple nodes within a preset future time period are determined.

[0044] Specifically, the prediction model includes a Long Short-Term Memory (LSTM) network model and an ARIMA algorithm model. For example, in an embodiment of this application, an integrated log analysis tool can be used to collect the resource usage curves and data access patterns of nodes over a recent period, including request frequency, concurrency, and hot data access patterns. Then, this time-series data is input into the trained LTM network model for prediction, outputting the probability and time window of node resource exhaustion or offline within a preset future time period. Simultaneously, a database is monitored to obtain historical data on node real-time load and data access patterns, and the ARIMA algorithm is used to train the model to predict node load and data access pattern information within a preset future time period.

[0045] Through the above solutions, this application embodiment uses a predictive model to make forward-looking predictions of node load trends and data access patterns in the future time period, so that the sharding strategy can respond in advance to business peaks or changes in resource pressure, avoiding performance bottlenecks and system instability caused by sudden traffic or concentrated access.

[0046] Step S120: Determine the dependency graph between the multiple shards based on the data access mode information.

[0047] The dependency graph includes dependency nodes and dependency edges. The dependency nodes indicate the shards, and the dependency edges indicate the data access dependencies between the shards. Each dependency edge also includes a dependency weight, which indicates the strength of the dependency between two shards.

[0048] For example, in the embodiments of this application, the dependency graph is a directed graph used to describe the data access dependencies between shards in a distributed database. A dependent node represents a shard, and a dependency edge represents a data access dependency from one shard to another; the direction of the edge indicates the direction of the dependency. That is, if a shard needs to access data from another shard during runtime, the path points from that shard to the dependent shard. The dependency weight is a numerical value that quantifies the strength of the dependency between two shards; a larger weight indicates a tighter dependency. Access frequency refers to the number of times a data item is read or written per unit of time. The number of dependency connections refers to the frequency of cross-node interactions between two shards, reflecting the activity level of data access between shards.

[0049] Optionally, in step S120, this embodiment of the application can determine directed edges to obtain a directed graph based on the dependency direction of the partition in the dependency graph, and detect cycles in the directed graph starting from each partition to identify cyclic dependencies. Specifically, directed edges are determined based on the dependency direction of the partition in the dependency graph to obtain a directed graph; starting from each partition, cycles in the directed graph are detected, and node states are marked, including unvisited, currently visited, and visited; if a currently visited node is detected, it is determined that a directed cycle exists in the directed graph, and the cyclic dependency is determined to exist. The cyclic dependency is used to indicate that the dependency relationship between multiple partitions forms a closed loop.

[0050] Optionally, in step S120, this embodiment of the application can identify cross-dependencies and their hot shards by detecting the frequency of inter-service calls and resource sharing. Specifically, by detecting the frequency of inter-service calls and resource sharing, if the resource sharing is greater than a preset sharing threshold, it is determined that there is a cross-dependency between the shards. Then, based on the frequency of inter-service calls, the shard frequently called by multiple upstream sources is identified as the hot shard. This cross-dependency is used to indicate that two or more shards depend on the same set of data.

[0051] Step S130: Determine the sharding strategy based on the node's lifecycle information, node load information, data access mode information, and dependency graph.

[0052] The sharding strategy is used to indicate which shards among the multiple shards are to be split or merged.

[0053] For example, embodiments of this application can predict the offline time of nodes based on node lifecycle information, and then determine the need to adjust the data sharding strategy. For instance, by calculating the lead time for each node's offline and the node's load status, combined with the start and end times of the node's lifecycle, it can be determined whether data shards need to be reallocated. Furthermore, it checks whether the data shards responsible for the node about to go offline can be migrated to other nodes before the node goes offline, and by continuously monitoring the load status of the target node, identifies possible bottleneck nodes and their data transmission paths, obtaining an impact analysis report on node load and data transmission. Then, based on the resource usage and load status of the new node, it is determined whether the new node needs to be allocated new data shards, or whether load balancing of existing shards is necessary.

[0054] As another example, embodiments of this application can dynamically calculate the load changes of each node based on the node's lifecycle information, assess the node's impact on data sharding, and obtain the sharding range, load distribution, and corresponding nodes that need adjustment. Furthermore, based on the load patterns of different nodes, the sharding range that needs to be migrated is determined, resulting in the need to adjust the sharding strategy. Simultaneously, based on real-time data of data access patterns, frequently accessed shards and low-load shards are identified, and shards that need to be split or merged are determined. The following embodiments will describe this part in detail.

[0055] It should be understood that, in the embodiments of this application, the sharding strategy refers to a scheme for adjusting data sharding based on the current operating status of the system and future predicted trends, including determining which shards need to be split, which shards need to be merged, and the target nodes for shard migration.

[0056] Optionally, in step S130, this application may also comprehensively consider multiple factors such as system load distribution, node lifecycle changes, and sharding resource requirements to formulate a sharding strategy. Specifically, embodiments of this application may first determine the current system load distribution status, node lifecycle changes, and sharding resource requirements based on the node load information and the data access mode information, and then determine the sharding strategy based on the load distribution status, node lifecycle changes, and resource requirements.

[0057] Step S140: Based on the dependency graph and the sharding strategy, perform sharding splitting or sharding merging operations on the multiple shards.

[0058] Specifically, in step S140, this embodiment of the application can perform splitting or merging operations on the corresponding shards according to the constructed dependency graph and the formulated sharding strategy. For example, before performing shard splitting or merging operations, the dependency relationships of related shards are identified according to the constructed dependency graph, and circular dependencies and cross dependencies between shards are marked, and circular dependency handling strategies and cross dependency handling strategies are formulated.

[0059] Optionally, embodiments of this application can prioritize shards with strong dependencies based on the strength of those dependencies. Furthermore, if splitting a shard would lead to interruption of the access path or disruption of dependencies, embodiments of this application can postpone the operation or select a different target node for migration. After performing shard splitting or merging operations, embodiments of this application can dynamically update the dependency graph and adjust data synchronization and access strategies. Based on the updated dependency graph, embodiments of this application can perform real-time monitoring of dependency changes, analyzing whether new dependency changes exist after each sharding operation through a monitoring system.

[0060] Optionally, in step S140, this embodiment of the application can determine the dependency mismatch risk between shards based on the dependency weights indicated in the dependency graph, and automatically adjust when the risk exceeds a threshold. Specifically, the dependency mismatch risk between shards is determined based on the dependency weights indicated in the dependency graph; if the dependency mismatch risk is greater than a preset risk threshold, at least one of the following adjustment measures is taken: adjusting the load of the shard; transferring part of the load to other nodes; optimizing the access path of the shard; migrating the shard to a more suitable node; merging or splitting two shards to reduce cross-shard dependencies.

[0061] In some embodiments of this application, the dependency mismatch risk between two shards can be determined based on the dependency strength between the two shards, the current load of the two shards, the communication delay between the two shards, and a preset adjustment coefficient. If the dependency mismatch risk between the shards exceeds a preset risk threshold, it indicates that the shard needs to be adjusted.

[0062] Optionally, in embodiments of this application, multiple processing strategies can be executed to break circular dependencies upon detection. Specifically, upon detection of a circular dependency, at least one of the following processing strategies is executed: abstracting a common interface or service to transform the circular dependency into a one-way dependency; replacing direct calls with asynchronous messages or an event bus; and resolving startup deadlock issues using the target mechanism within the framework. This application, by accurately identifying circular dependencies between shards and formulating differentiated processing strategies, transforms bidirectional call relationships into one-way call relationships, breaks circular dependencies, and significantly reduces the coupling between services.

[0063] Optionally, in embodiments of this application, hotspot shards can be identified and targeted processing strategies can be implemented upon detection of cross-dependencies. Specifically, upon detection of a cross-dependency, hotspot shards within the cross-dependency are identified; at least one of the following processing strategies is implemented for the hotspot shard: cache optimization is performed for the hotspot shard; the data of the hotspot shard is partitioned and maintained locally within a module. By accurately identifying cross-dependency hotspots between shards and taking targeted governance measures, and by introducing a caching mechanism and module-level local data maintenance, this application significantly reduces access latency and resource contention caused by cross-dependencies, and reduces the resource conflict rate between shards.

[0064] Optionally, in step S140, this embodiment of the application can prioritize the fragments to be split or merged based on the dependency strength in the dependency graph. Specifically, based on the dependency strength in the dependency graph, the fragments to be split or merged as indicated in the sharding strategy are prioritized, wherein fragments with higher dependency strength have higher priority than fragments with lower dependency strength; according to the sharding strategy, shard splitting or shard merging operations are performed on the multiple fragments, including: performing shard splitting or shard merging operations according to the sharding strategy and the result of the priority ranking. By prioritizing fragments with strong dependencies, this application ensures that highly coupled fragments are prioritized during adjustments, effectively reducing the risk of data access anomalies or performance bottlenecks caused by fragment adjustments.

[0065] By employing the method 100 described above, this embodiment of the application constructs a dependency graph between shards and quantifies the dependency strength. This allows for accurate identification of dependencies between relevant shards before performing shard splitting or merging operations, preventing access path interruptions or dependency disruptions caused by splitting, thus enhancing the modularity and maintainability of the system. Simultaneously, by dynamically formulating sharding strategies based on node load information and data access pattern information, the number and size of shards become more reasonable, reducing frequent re-sharding operations and improving overall system performance.

[0066] Furthermore, this embodiment of the application also predicts the load trends and data access patterns of nodes over future time periods, considering the dynamic evolution trends of data distribution and access load during the sharding planning stage. This makes the determined number and size of shards more reasonable, thereby reducing frequent re-sharding operations during system operation and improving overall system performance. Moreover, this embodiment of the application also constructs a dependency graph between shards and combines it with dependency strength analysis to accurately identify circular and cross dependencies between shards. Differentiated processing strategies are formulated for different dependency patterns, significantly reducing the coupling between services and enhancing the modularity and maintainability of the system.

[0067] The following describes a detailed embodiment of method 100 with reference to the accompanying drawings. Figure 2 This diagram illustrates a flowchart 200 of a fast data fragmentation method provided in an embodiment of this application. Figure 2 As shown, this process flow 200 includes steps S201 to S205. The contents of steps S201 to S205 are described in detail below.

[0068] Step S201: Obtain the lifecycle information of each node through the monitoring and logging mechanism of the distributed database sharding system.

[0069] In the embodiments of this application, an integrated node lifecycle management module can be used to monitor the status of nodes in real time and record data load and system load information when nodes go online and offline. The status can include online, offline, load, and resource consumption. Furthermore, timestamps and status markers are used to mark the start and end of each node's lifecycle, and the node's resource consumption, load data, network bandwidth, and memory usage are recorded.

[0070] Furthermore, the embodiments of this application can record key events in the node lifecycle in real time through log analysis tools, and perform statistical analysis on the node's workload, resource consumption and data access status during the lifecycle to obtain the node's resource consumption status and data access patterns at different lifecycle stages. Key events include going online, going offline and status changes.

[0071] For example, in a distributed database cluster, server Node-07 officially went online on January 15, 2020, at 08:00:00, and its status became online. At the time of going online, the initial data load of this node was 150GB, the overall system load was 65%, with Node-07's CPU utilization at 42%, memory utilization at 58%, and peak network bandwidth usage at 180Mbps. Throughout its lifecycle, Node-07's status underwent several changes. In particular, on January 20, 2020, at 10:30:00, the system detected an increase in the node's load, reaching 89%, and marked its status as high load. At this time, its average data access requests reached 4200 times / minute. Log records show that during this period, the I / O load was abnormally frequent, the peak bandwidth also rose to 310Mbps, and the memory utilization exceeded 85%. Subsequently, at 02:15:00 on January 22, 2020, Node-07 was manually taken offline due to a maintenance plan. The system recorded its status as offline, with the termination timestamp being this time. Before being taken offline, the load dropped to 48%, but 175GB of persistent data remained awaiting transfer. The entire lifecycle, from launch to shutdown, lasted 7 days, 18 hours, and 15 minutes, and was recorded as the lifecycle length. During this period, the log analysis module continuously recorded key events of Node-07, including launch, load changes, peak resource consumption, abnormal states, and final shutdown. Based on this data, statistical analysis tools determined that the average CPU utilization during its lifecycle was 61.2%, memory utilization was 73.5%, and average bandwidth usage was 210Mbps. The system also identified the most intensive data access patterns between day 5 and day 6, accounting for 34% of the total access volume. Ultimately, the system used this information in the lifecycle assessment report to provide data support for future node resource scheduling, load balancing, and high availability strategy optimization.

[0072] Step S202: Based on the node's lifecycle information, predict the node's offline time and determine the need to adjust the data sharding strategy.

[0073] In this step, this embodiment of the application can obtain the current resource usage, load mode, and data access mode of each node based on node lifecycle information, and use a Long Short-Term Memory (LSTM) network for model training to predict the node's offline time. Then, by calculating the lead time for each node's offline and the node's load status, combined with the start and end times of the node's lifecycle, it is determined whether data shards need to be reallocated. It checks whether the data shards responsible for the node about to go offline can be migrated to other nodes before the node goes offline, and by continuously monitoring the load status of the target node, it identifies possible bottleneck nodes and their data transmission paths, obtaining an impact analysis report on node load and data transmission. Furthermore, based on the resource usage and load status of the new node, it is determined whether the new node needs to be allocated new data shards or whether load balancing of existing shards is necessary. Based on the node's lifecycle information, the load change of each node is dynamically calculated, and the impact of that node on data shards is assessed to obtain the shard range, load allocation, and corresponding nodes that need adjustment. Finally, based on the load modes of different nodes, the range of shards that need to be migrated is determined, resulting in the need to adjust the sharding strategy.

[0074] For example, in a distributed database system with 10 nodes, node Node-04 exhibited a significant resource degradation trend in the recent monitoring period. According to the node lifecycle management module, this node has been running for 35 days and 6 hours since it went online at 00:00 on January 1, 2020. System logs show that in the past 10 days, the node's average CPU utilization increased from 68.2% to 93.5%, memory utilization increased from 74.1% to 96.8%, and network bandwidth usage continuously approached the system's set threshold of 500Mbps, reaching a peak of 495Mbps. Through integrated log analysis tools, the system collected Node-04's resource usage curves and data access patterns over the past 30 days, including request frequency, concurrency, and hot data access patterns, and input this time-series data into a trained Long Short-Term Memory (LSTM) network model for prediction. The model output indicates that Node-04 is highly likely to experience resource exhaustion or instability and go offline within the next 48 hours, with a predicted offline window from 02:00 to 06:00 on August 6, 2020, and a prediction confidence level of 87.3%. Based on this result, the system calculated the lead time for Node-04's offline, i.e., the remaining window before the predicted offline time, which is approximately 36 hours. At this time, the system scheduler calls the node mapping table and finds that Node-04 currently hosts three data shards, namely Shard-3, Shard-8, and Shard-11. Among them, Shard-8 is the core shard of a high-frequency service, with an access request frequency of 8500 times / minute, accounting for 14.7% of the total system requests. The system assesses the real-time load of all nodes and identifies Node-06 and Node-09 as resource-rich nodes, with CPU utilization rates of 38.2% and 41.7%, respectively, memory utilization rates below 50%, and network bandwidth far from saturated. Therefore, Shard-8 was migrated to Node-06 first, while Shard-3 and Shard-11, which had lower access pressure, were migrated to Node-09 and Node-01 respectively, to complete the shard redistribution as quickly as possible and avoid data unavailability or access interruption caused by the offline status of Node-04. Meanwhile, the load growth of the target nodes, especially Node-06, was continuously monitored. In the first hour after Shard-8 was deployed and enabled, the bandwidth usage of Node-06 increased from 220Mbps to 390Mbps, but still did not reach the system warning threshold of 450Mbps, indicating that the migration operation did not create a new bottleneck. In addition, the system detected a newly launched node, Node-11, whose current CPU usage was only 22.6% and memory usage was 19.5%, indicating it was completely idle.Based on the current system access patterns and node lifecycle predictions, the load balancing module determines that Node-11 can handle some of the shard load from Node-07 to achieve overall system load balancing. Ultimately, the system smoothly migrates Shard-5 from Node-07 to Node-11, which is expected to reduce Node-07's CPU load by approximately 9%. After these operations, the system outputs a shard redistribution strategy adjustment report, clearly indicating the migration paths of the three shards, the risk assessment of predicted offline nodes, the identification of bottleneck nodes, the load status of the target nodes after migration, and whether the new nodes are capable of supporting shards.

[0075] Step S203: Based on real-time data of data access patterns, identify high-frequency access shards and low-load shards, and determine which shards need to be split or merged.

[0076] In this step, embodiments of this application can obtain the data access volume, frequency, and data access patterns of each node by configuring a log collection system, monitoring software, or database to obtain real-time data on data access patterns, including access hotspots and coldspots. Then, through statistical and trend analysis of historical data access patterns, the access frequency of each data item is calculated, the access fluctuation range is calculated, and changes in access patterns are detected to determine whether there are shards with excessively high loads or shards with low loads. Next, by setting access frequency and load thresholds, the changing trends of data access volume and frequency are monitored periodically to identify shards with high-frequency access and shards with low loads, and to determine whether to trigger shard merging or splitting operations. If the data access volume or load of a node approaches the set threshold, a sharding splitting operation is triggered, and a load balancing algorithm is used to cut the data shards with high loads into multiple sub-shards, and the load is optimized by adjusting the data distribution. If the access volume of a shard gradually decreases and it is in a low-load state, a shard merging operation is performed. Based on the data access pattern, access frequency and load status, it is determined whether it needs to be merged with other shards, and the shard with the lowest load is selected for merging to optimize resource allocation.

[0077] For example, in a distributed database system, node Node-02 hosts three shards: Shard-A, Shard-B, and Shard-C. The system deploys a log collection system and monitoring software, integrated with the database in real time, to collect access data for each shard, including access volume, access frequency, and data access patterns, including high-frequency reads and periodic batch writes. Over the past seven consecutive days, the system detected a continuous upward trend in data access volume for Shard-A. Historical data analysis shows that the average daily access requests for this shard increased from 28,000 to 86,000, and the fluctuation range of access frequency increased from ±4,000 to ±12,000, indicating that this shard not only experiences intensive access but also significant load fluctuations during peak periods. The system sets an access frequency threshold of 80,000 times per day for shards; therefore, once this value is exceeded, the shard is identified as high-load and requires immediate evaluation for splitting. The system confirmed that the current access to Shard-A had exceeded the threshold, and the monitoring software observed that Node-02's CPU utilization reached 92% and memory usage reached 89%. According to the load balancing strategy, the system performed an automatic sharding operation, splitting Shard-A into two sub-shards, Shard-A1 and Shard-A2, which were migrated to the less loaded nodes Node-05 and Node-07 respectively, to alleviate the burden on Node-02. Before the split, Shard-A alone consumed 56% of Node-02's total I / O capacity; after the split, this proportion dropped to 21%, successfully achieving load sharing. Meanwhile, Shards Shard-D and Shard-E running on Node-03 were detected as cold shards. Log data showed that their access frequency had been below the system's set low-load threshold of 5,000 times / day for two consecutive weeks. Shard-E had an average of only 1,200 accesses per day, while Shard-D was slightly higher at 2,300, and the resource utilization of these two shards remained consistently below 15%. The system determined that Shard-D and Shard-E were in a state of resource waste, thus triggering the shard merging mechanism. During the merging evaluation process, the system analyzed the data structure compatibility and access pattern similarity of Shard-D and Shard-E. Both were primarily low-frequency reads, with a total data volume of 22GB, far below the maximum capacity of 50GB for a single shard. Ultimately, Shard-D and Shard-E were merged into a new shard, Shard-DE, which remained on Node-03. After the merge, the CPU utilization of Node-03 decreased from 31% to 26%, freeing up approximately 380MB of memory and making room for other more active shards.This complete process starts with real-time monitoring data, combines historical access patterns, threshold strategies, and load balancing algorithms, automatically identifies high-frequency hotspots and coldspots, and adopts splitting and merging strategies respectively. Ultimately, it improves the overall resource utilization of the system, reduces the risk of performance bottlenecks, and also reduces the redundant overhead of cold data shards.

[0078] Step S204: Based on the real-time load status and data access patterns of the nodes, predict the node load and data access pattern information within a preset time period in the future, determine the number and size of shards, formulate a sharding strategy, and construct a dependency graph between shards. Based on the strength of the dependency relationship between shards, prioritize the processing of shards with strong dependencies.

[0079] Figure 3 This diagram illustrates the flow 300 of the fragmentation processing procedure based on inter-fragment dependencies in step 104. Figure 3 As shown, the process includes steps S301 to S308.

[0080] In step S301, this embodiment of the application can obtain the real-time load status and data access pattern of each node through a system monitoring tool and store them in a data sharding monitoring database. Then, in step S302, this embodiment of the application can obtain historical data on the real-time load status and data access pattern of the nodes through the data sharding monitoring database, and use the ARIMA algorithm to train a model to predict the node load and data access pattern information within a preset future time period.

[0081] Next, in step S303, based on the predicted node load and data access pattern information within a preset future time period, combined with the current system load distribution, node lifecycle changes, and shard resource requirements, the number and size of shards are determined. Furthermore, corresponding to step S103, in step S304, this embodiment formulates a sharding strategy based on data access hotspots, automatically splitting or merging shards. Corresponding to step S102, in step S305, this embodiment constructs a dependency graph between shards based on their data access patterns, identifying the dependencies between shards.

[0082] For example, the formula for dependency strength can be as follows:

[0083] Furthermore, it can be used to calculate the dependency strength between fragment i and fragment j. .in, and Let i and j represent the frequency of access to a certain data item k, respectively. It is the number of dependency connections between shard i and shard j, representing the frequency of cross-node interactions between the two shards.

[0084] In the dependency graph, each shard is a node, edges represent cross-shard data access dependencies, and weights in the dependency graph represent the dependency strength between two shards. Therefore, corresponding to step S105, in step S306, this embodiment can identify the dependencies between related shards based on the constructed dependency graph before performing shard splitting or merging operations, and mark circular and cross dependencies between shards, formulating circular dependency handling strategies and cross dependency handling strategies. Furthermore, in step S307, this embodiment can prioritize shards with strong dependencies based on the strength of the dependencies between shards.

[0085] If splitting a shard would cause access path interruption or dependency disruption, the operation is postponed or a different target node is selected for migration. After shard splitting or merging, the dependency graph is dynamically updated, and data synchronization and access strategies are adjusted. Based on the updated dependency graph, real-time monitoring of dependency changes is performed, and the monitoring system analyzes whether new dependency changes occur after each sharding operation. Then, in step S308, the dependency mismatch risk between shards is determined based on the dependency strength between shards and the current load. If the detected dependency mismatch risk exceeds a preset risk threshold, a dependency mismatch alarm is triggered, and the status of that shard is adjusted.

[0086] For example, the formula for assessing the risk of dependency mismatch is as follows:

[0087] Furthermore, the dependency mismatch risk between shard i and shard j can be determined. .in, The strength of the dependency between fragment i and fragment j and The current load of shards i and j is given by α, which is an adjustment coefficient used to adjust the impact of load differences on risk. It is obtained by fitting historical data. This represents the communication latency between shards i and j, reflecting the latency of cross-shard access. If the risk of dependency mismatch between shards exceeds a preset risk threshold, it indicates that the shard needs adjustment.

[0088] For example, the load status of two shards, S1 and S2, and the strength of their dependency relationship are obtained in real time through system monitoring tools. If the dependency strength C12 between shards S1 and S2 is 0.8, it indicates a strong dependency between the two shards. The current load L1 of shard S1 is 75%, while the current load L2 of shard S2 is 50%. Furthermore, an adjustment coefficient α is set to 0.5, indicating that the load difference has a certain moderating effect on the risk. The system also learns through monitoring tools that the communication latency σ12 between shards S1 and S2 is 0.2, reflecting the data access latency between the two shards. The dependency mismatch risk assessment formula is then used. The calculated dependency mismatch risk R12 between fragments S1 and S2 is 33.33. To mitigate the risk of dependency mismatch between shard i and shard j, The strength of the dependency between fragment i and fragment j and The current load of shards i and j is given by α, which is an adjustment coefficient used to adjust the impact of load differences on risk. It is obtained by fitting historical data. This represents the communication latency between shards i and j, reflecting the latency of cross-shard access. If the system's preset risk threshold is 30, and the calculated risk value of 33.33 is greater than the preset risk threshold of 30, this indicates a high risk of mismatch in the dependency relationship between shards S1 and S2, requiring adjustment.

[0089] For example, in a distributed database, there are three shards, S1, S2, and S3, stored on different nodes. Each shard has a different data access pattern, and these shards are interdependent. By monitoring the load of each node in real time, if node 1's load is 80%, node 2's load is 60%, and node 3's load is 90%, the system also obtains data access pattern information in real time, such as shard S1 accessing data item k 100 times / second, shard S2 accessing data item k 150 times / second, and shard S3 accessing data item k 50 times / second. Then, using this data, the ARIMA algorithm is used to predict the load and data access patterns for future time periods, resulting in the predicted node load for the next hour: node 1 will reach 95%, node 2 will remain at 60%, and node 3 will decrease to 75%.

[0090] Furthermore, based on this predictive information, the system analyzes the existing load distribution to determine whether the number and size of shards need to be adjusted. According to the predicted load, the system can determine that node 1, with its high load, needs to be split or migrated, while node 3 may not require immediate processing. A dependency graph is constructed based on data access hotspots and dependencies between shards. Each shard is a node in the graph, edges represent cross-shard data access dependencies, and weights in the dependency graph represent the strength of the dependency between two shards. In the dependency graph, shards S1 and S2 have a strong dependency, especially with a high access frequency for data item k. Shard S3 has a relatively weak dependency with the other two shards. Using the dependency strength formula, the dependency strength between S1 and S2 is calculated to be C12 = 0.75, while the dependency strength between S1 and S3 is C13 = 0.3, and the dependency strength between S2 and S3 is C23 = 0.4.

[0091] Based on the calculated dependency strength, the system finds a strong dependency between S1 and S2. Therefore, when splitting or merging shards, these dependencies should be prioritized to avoid data access interruptions caused by splitting or migrating. If splitting S1 affects the data access path of S2, the system will postpone the splitting operation or choose to migrate the shard to other nodes with lower load. Assume the system decides to migrate shard S3 to node 2 with lower load because its dependency is weak and node 2 has a lower load. After splitting or merging shards, the dependency graph is dynamically updated, and real-time monitoring of dependency changes for each shard begins. If splitting a shard causes a dependency mismatch, for example, causing an interruption in the data access path between S1 and S2, the system will trigger a dependency mismatch alarm and take corresponding measures to fix the problem, ensuring system stability. Assume that during monitoring, the system detects a change in the dependency between S1 and S2. Based on the dependency strength between shards and the current load, the calculated dependency mismatch risk value is 33.33, exceeding the preset risk threshold of 30. At this point, the system will trigger an alarm and reassess whether further adjustments to S1 or S2 are needed to avoid potential performance bottlenecks. Adjustments may include adjusting the load on one of the shards to reduce pressure on shard S1, or transferring some load to other nodes. If the load difference is significant and communication latency is high, consider optimizing the shard access path or migrating the shard to a more suitable node to reduce the risk of dependency mismatch, or merging or splitting the two shards to reduce cross-shard dependencies.

[0092] In some other embodiments of this application, a directed graph can be obtained by drawing a directed edge based on the dependency relationship graph between the shards. Then, a depth-first search is used to detect cycles in the graph. A depth-first search traversal is performed starting from each shard, and the node state is marked, including unvisited, currently visited, and visited. If a currently visited node exists during the traversal, it is determined that a directed cycle exists in the directed graph, indicating a circular dependency between the shards. A circular dependency refers to a closed-loop dependency relationship formed between a group of shards.

[0093] In other embodiments of this application, APM tools or log analysis systems can be used to monitor the frequency of calls between services and the degree of resource sharing. If the degree of sharing exceeds a preset threshold, it is determined that there is a cross dependency between shards. Cross dependency refers to two or more shards depending on the same set of data, or multiple services accessing shared resources at runtime. If a shard is frequently called by multiple upstream services, it is determined to be a cross dependency hotspot.

[0094] Furthermore, based on the identification results of circular dependencies and cross dependencies, embodiments of this application can formulate circular dependency handling strategies and cross dependency handling strategies. Circular dependency handling strategies include one or more of the following: abstracting public interfaces or services, making dependencies one-way, using asynchronous messages or event buses to replace direct calls, and using the @Lazy mechanism in the Spring and Guice frameworks to solve startup deadlock problems. Cross dependency handling strategies include one or more of the following: using Redis, Memcached, or local caching for hot resources, and dividing cross dependency hotspots into local maintenance within modules.

[0095] For example, in a distributed database architecture, the system contains seven logical shards, designated Shard-A to Shard-G. The platform's technical team constructed a dependency graph based on the dependencies between the shards. Each directed edge represents a shard's dependence on the data or state of another shard during runtime; for instance, Shard-A → Shard-B indicates that A needs to access B's data index when processing a query. After constructing the dependency graph, the development team used a depth-first search algorithm to traverse each shard, marking each shard as unvisited, currently being accessed, or already accessed. When traversing the path Shard-C → Shard-E → Shard-F → Shard-C, the system detected that F depends on C, and C is currently being accessed. This indicates that a closed loop is formed between these three shards. Therefore, it was determined that a circular dependency exists between Shard-C, Shard-E, and Shard-F, and the system may fail to initialize or split any shard due to dependency conflicts. First, analyzing the dependencies of the three shards revealed that they mutually referenced user activity rating data. Therefore, this rating logic was extracted into an independent rating service module. All shards only need to obtain rating results through the asynchronous event message bus Kafka, eliminating mutual calls and transforming the original bidirectional call relationship into a unidirectional event subscription relationship, completely breaking the circular dependency. At the framework level, the system also used Spring's `@Lazy` annotation for Shard-C initialization injection to avoid triggering dependency resolution immediately during service startup. Furthermore, real-time tracing of the inter-service call chain was performed using integrated APM tools such as SkyWalking. This revealed that Shard-B is a frequently accessed common shard by Shard-A, Shard-D, and Shard-G, being called an average of 890 times per minute, 3.5 times the platform average of 250 times per minute. Further analysis showed that these shards jointly depend on Shard-B's user session log data, forming a typical cross-dependency hotspot. The resource sharing index given by the APM system was 0.78, higher than the set sharing threshold of 0.6, indicating that Shard-B is a high-risk dependency point. To alleviate performance pressure and reduce coupling, a Redis cache was introduced for the hot data of this shard, storing the most active session data from the last 24 hours in the local node's memory to reduce frequent remote access. Simultaneously, the log structure was rationally partitioned, separating frequently analyzed user click logs from other behavior logs, and synchronizing copies of user click data to the Shard-A local cache to achieve module-level local data maintenance, thereby significantly reducing access latency and resource contention caused by cross-dependencies. After a week of operation and observation, the overall system response time decreased from an average of 380ms to 220ms, the resource conflict rate between shards decreased by 47%, and the service initialization success rate increased to 100%.Therefore, the strategies for identifying and decoupling circular and cross dependencies are summarized into a template and applied to other business segments of the system.

[0096] Step S205: Obtain performance monitoring data of each node after data sharding adjustment through performance monitoring tools, evaluate the sharding optimization effect, and adjust the sharding strategy.

[0097] In this step, the embodiments of this application can obtain performance monitoring data of each node after data sharding adjustment through performance monitoring tools, including load, response time, and storage usage data. The differences in performance monitoring data before and after adjustment are compared to evaluate the effectiveness of the current sharding optimization strategy. If the system achieves the expected goals in terms of load balancing and response time, this sharding strategy is applied to the entire system. If the load between some nodes is still unbalanced or some shards still have excessive load, the sharding strategy needs to be re-evaluated. Using data obtained from the monitoring system, the load fluctuations and data access patterns of different nodes are checked periodically to identify shards with excellent sharding adjustment effects and those with poor effects. The sharding strategy and node scheduling scheme are adjusted through the intelligent scheduling system Kubernetes. Furthermore, based on the load fluctuations of different nodes throughout their lifecycle, combined with the access frequency and data volume of the shards, the sharding strategy is re-optimized, and the number of shards, the location of shards, and the distribution of replicas are dynamically adjusted as the node lifecycle changes. By monitoring the database through data sharding, we can obtain historical data on the real-time load status and data access patterns of nodes. We can then use the ARIMA algorithm to train a model, predict future changes in node load, and adjust the sharding strategy based on the prediction results to avoid performance bottlenecks caused by excessively high or low node loads.

[0098] For example, in a large distributed database system, the system administrator performed a sharding optimization operation on July 1, 2020, primarily targeting the redistribution of high-load shards on Node-03, Node-06, and Node-09. Specifically, the overloaded shard Shard-Q on Node-06 was split into Shard-Q1 and Shard-Q2, which were then migrated to Node-07 and Node-08 respectively. Simultaneously, the two shards Shard-M and Shard-N on Node-03, which experienced continuously decreasing access, were merged to form a new Shard-MN shard, which was retained on the original node. After the optimization, the operations team compared the performance metrics of each node before and after the adjustment using Prometheus and Grafana monitoring dashboards for a week. Before the adjustment, Node-06 had an average CPU utilization of 92.3% and an average response time of 380ms, far exceeding the system's set health thresholds of CPU <80% and response time <250ms. After a week of adjustment, Node-06's average CPU load dropped to 66.7%, and the response time stabilized at around 205ms, demonstrating a significant performance improvement. Meanwhile, the merged Shard-MN shard on Node-03 had an average daily access frequency of 3900 times, compared to the two original shards' average daily total access frequency of 4100 times, indicating a stable access trend. Node-03's storage usage decreased from 86GB to 58GB, and CPU utilization dropped from 47% to 31%, demonstrating that the merging strategy effectively freed up system resources. However, the evaluation also revealed that while Node-09 initially handled the split Shard-Q2 shard, it also hosted another hot shard, Shard-L. This caused its CPU utilization to surge rapidly within a week after optimization, rising from 63% to 91%, and its response time increased from 215ms to 360ms. Based on this, the Kubernetes scheduling system determined that this node had become a new bottleneck, triggering the automatic scheduling mechanism to migrate Shard-Q2 to the more idle Node-11, which has a CPU utilization of only 22%, storage usage of 35GB, and bandwidth utilization of less than 40%. Further strategy optimization involved analyzing the time series of load and access data for each node over the past 30 days using the accessed data shard monitoring database. An ARIMA prediction model was then used to model the node load trend for the next 7 days. The model predicted that Node-08's load would remain stable between 45% and 53%, suitable for continuing to handle the current shard. However, if Node-09 did not undergo shard migration, its load would exceed 95% within the next 5 days, which was expected to severely impact performance.Based on the above data, the system ultimately generated an automated adjustment recommendation report, suggesting migrating Shard-Q2 from Node-09 to Node-11 and re-partitioning the replica distribution structure of some less frequently used shards to further improve resource utilization efficiency. The system comprehensively evaluated the monitoring effects and prediction results of these adjustments, finding that 70% of the sharding adjustments met expectations in terms of resource consumption and response performance, while the remaining 30% posed a risk of new bottlenecks and required continued dynamic adjustment. This demonstrates that through systematic analysis and prediction of performance data, combined with Kubernetes' scheduling mechanism, the system not only evaluated the actual effectiveness of the current sharding strategy but also dynamically adjusted the sharding distribution strategy based on lifecycle load trends, effectively avoiding new performance bottlenecks and resource unevenness issues.

[0099] Through the above embodiments, based on the real-time load status and data access patterns of nodes, the node load and data access pattern information within a preset time period can be predicted, the number and size of shards can be determined, and a sharding strategy can be formulated. Furthermore, a dependency graph between shards is constructed, and shards with strong dependencies are prioritized based on the strength of the dependencies. Moreover, based on the constructed dependency graph, this embodiment can identify the dependencies between related shards, mark circular and cross dependencies between shards, and formulate circular dependency handling strategies and cross dependency handling strategies. In addition, based on the strength of the dependencies between shards and the current load, this embodiment can determine the risk of dependency mismatch between shards and adjust the state of the shard.

[0100] Furthermore, this embodiment predicts the load trends and data access patterns of nodes over future time periods, considering the dynamic evolution of data distribution and access load during the sharding planning stage. This makes the determined number and size of shards more reasonable, thereby reducing frequent re-sharding operations during system operation and improving overall system performance. Moreover, by constructing a dependency graph between shards and combining it with dependency strength analysis, accurate identification of circular and cross dependencies between shards is achieved. Differentiated processing strategies are formulated for different dependency patterns, significantly reducing the coupling between services and enhancing the modularity and maintainability of the system. In addition, this embodiment, through a joint judgment mechanism of dependency strength and current node load, can identify potential dependency mismatch risks between shards and automatically adjust the operating status of high-risk shards, preventing data access anomalies or performance bottlenecks caused by inconsistent dependencies.

[0101] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0102] This application provides a readable storage medium containing instructions that, when executed by an electronic device, cause the electronic device to perform the technical solution described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0103] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0104] The hardware module of this application is described below, which can be used to implement the aforementioned method 100 and Figures 2 to 3 The content shown.

[0105] Now for reference Figure 4 The diagram shows a block diagram of a device 400 according to one embodiment of this application. Device 400 may include one or more processors 401 coupled to a controller hub 403. In at least one embodiment, the controller hub 403 communicates with the processor 401 via a multi-branch bus such as a front side bus (FSB), a point-to-point interface such as a quickpath interconnect (QPI), or a similar connection 410. The processor 401 executes instructions controlling general types of data processing operations. In one embodiment, the controller hub 403 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0106] Device 400 may also include a coprocessor 402 and a memory 404 coupled to a controller hub 403. Alternatively, one or both of the memory and the GMCH may be integrated within the processor, with memory 404 and coprocessor 402 directly coupled to processor 401 and controller hub 403, which resides on a single chip with the IOH. Memory 404 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, coprocessor 402 is a dedicated processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of coprocessor 402 are indicated by dashed lines. Figure 4 middle.

[0107] Memory 404, as a computer-readable storage medium, may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 404 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0108] In one embodiment, device 400 may further include a network interface controller (NIC) 406. NIC 406 may include a transceiver for providing a radio interface to device 400, thereby enabling communication with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, NIC 406 may be integrated with other components of device 400. NIC 406 can implement the functionality of the communication unit in the above embodiments.

[0109] Device 400 may further include input / output (I / O) device 405. I / O 405 may include: a user interface designed to enable a user to interact with device 400; a peripheral component interface designed to enable peripheral components to also interact with device 400; and / or sensors designed to determine environmental conditions and / or location information related to device 400.

[0110] It is worth noting that, Figure 4 This is merely an example. That is, although... Figure 4 The diagram shows that device 400 includes multiple devices such as processor 401, controller hub 403, and memory 404. However, in actual applications, devices using the methods of this application may include only a portion of the devices in device 400. For example, it may include only processor 401 and NIC 406. Figure 4 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 404, which is a computer-readable storage medium, stores instructions that, when executed on a computer, cause the device 400 to perform the methods according to the above embodiments. Specific details can be found in the methods of the above embodiments, and will not be repeated here.

[0111] Now for reference Figure 5 The diagram shown is a block diagram of a system-on-chip (SoC) 500 according to an embodiment of this application. Figure 5 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 5 In this SoC 500, the following are included: an interconnect unit 550 coupled to an application processor 510; a system proxy unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more coprocessors 520, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 530; and a direct memory access (DMA) unit 560. In one embodiment, the coprocessor 520 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0112] The static random-access memory (SRAM) cell 530 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 500 to perform the method according to the above embodiments, as detailed in the methods described above, which will not be repeated here.

[0113] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0114] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0115] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0116] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0117] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0118] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0119] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A method for fast data sharding in a distributed database environment, characterized in that, include: Obtain node lifecycle information, node load information, and data access mode information for multiple nodes, wherein the node lifecycle information is used to predict the offline time of the nodes; Based on the data access pattern information, a dependency graph is determined among multiple shards deployed on the multiple nodes. The dependency graph includes dependency nodes and dependency edges. The dependency nodes are used to indicate the shards, and the dependency edges are used to indicate the data access dependencies between the shards. Each dependency edge also includes a dependency weight, which is used to indicate the strength of the dependency relationship between two shards. Based on the node lifecycle information, the node load information, the data access mode information, and the dependency graph, a sharding strategy is determined, which is used to indicate which shards among the plurality of shards are to be split or merged. Based on the dependency graph and the sharding strategy, perform sharding splitting or sharding merging operations on the multiple shards.

2. The method according to claim 1, characterized in that, The method further includes: Based on the dependency strength in the dependency graph, the shards to be split or merged as indicated in the sharding strategy are prioritized, wherein shards with higher dependency strength have higher priority than shards with lower dependency strength. The step of performing sharding splitting or sharding merging operations on the multiple shards according to the sharding strategy includes: Based on the sharding strategy and the result of the priority sorting, perform shard splitting or shard merging operations.

3. The method according to claim 1 or 2, characterized in that, The strength of the dependency relationship is determined based on the frequency of access to the same data item by the two shards and the number of dependency connections between the two shards.

4. The method according to claim 1 or 2, characterized in that, The dependencies between the shards include circular dependencies, which indicate that the dependencies between multiple shards form a closed loop. Before performing the shard splitting or shard merging operation indicated in the sharding strategy on the multiple shards, the method further includes: If the circular dependency is detected, at least one of the following processing strategies will be executed: Abstract out public interfaces or services to transform circular dependencies into unidirectional dependencies; Use asynchronous messages or event buses instead of direct calls; Use the target mechanism in the framework to solve the startup deadlock problem.

5. The method according to claim 4, characterized in that, The method further includes: Based on the dependency directions of the final partitions in the dependency graph, directed edges are determined to obtain a directed graph; Starting from each of the aforementioned fragments, loops in the directed graph are detected, and node states are marked, including unvisited, being visited, and visited. If a node being visited is detected, it is determined that a directed cycle exists in the directed graph, and the circular dependency is also determined.

6. The method according to claim 1 or 2, characterized in that, The dependencies between the shards include cross dependencies, which indicate that two or more shards depend on the same set of data. Before performing the shard splitting or shard merging operation indicated in the sharding strategy on the multiple shards, the method further includes: If the cross-dependency is detected, the hotspot fragments in the cross-dependency are determined; At least one of the following processing strategies will be applied to the hotspot shards: For the aforementioned hotspot shards, cache optimization is performed; The data from the hotspot shards is divided and maintained locally within the module.

7. The method according to claim 6, characterized in that, The method further includes: Detect the frequency of calls between services and the degree of resource sharing; If the resource sharing degree is greater than a preset sharing degree threshold, it is determined that there is a cross dependency between the fragments; Based on the frequency of inter-service calls, the shards that are frequently called by multiple upstream services are identified as the hot shards.

8. The method according to claim 1 or 2, characterized in that, The acquisition of node load information and data access mode information of multiple nodes includes: Obtain the real-time load status and data access patterns of the multiple nodes in the distributed database; Based on the real-time load status and data access patterns of the multiple nodes and the pre-trained prediction model, determine the node load information and data access pattern information of the multiple nodes within a future preset time period.

9. The method according to claim 8, characterized in that, The step of determining the sharding strategy based on the node load information and the data access pattern information includes: Based on the node load information and the data access mode information, determine the current system load distribution status, node lifecycle changes, and sharding resource requirements; The sharding strategy is determined based on the load distribution status, the node lifecycle changes, and the resource requirements.

10. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the dependency weights indicated in the dependency graph, determine the risk of dependency mismatch between the fragments; If the risk of dependency mismatch exceeds a preset risk threshold, at least one of the following adjustment measures shall be implemented: Adjust the load on the shards; Transfer some of the load to other nodes; Optimize the access path for shards; Migrate the shards to a more suitable node; Merge or split two shards to reduce cross-shard dependencies.

11. The method according to claim 10, characterized in that, Based on the dependency weights indicated in the dependency graph, determine the risk of dependency mismatch between the fragments, including: The dependency mismatch risk between the two shards is determined based on the dependency strength between the two shards, the current load of the two shards, the communication delay between the two shards, and a preset adjustment coefficient, wherein the adjustment coefficient is used to adjust the degree of influence of load difference on the dependency mismatch risk.

12. An electronic device, characterized in that, It includes one or more processors; one or more memories; said one or more memories storing one or more computer programs, said one or more computer programs including instructions that, when executed by said one or more processors, cause the method as described in any one of claims 1 to 11 to be performed.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the method as described in any one of claims 1 to 11 to be performed.

14. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 11.

Images