Database load optimization method and device based on dynamic fragmentation, equipment and medium

Abstract

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a database load optimization method based on dynamic fragmentation, which comprises the following steps: establishing an association relationship between fragments and storage nodes by configuring basic logic fragments and dynamic logic fragments; monitoring a data access record, identifying hotspot data and marking the hotspot data as a to-be-migrated data block; when the fragment load exceeds a threshold value, migrating the high-load data block to the dynamic logic fragment; and when the access frequency is reduced, moving the low-load data block back to the basic logic fragment. According to the method, through configuration and management of the dynamic logic fragments, in combination with real-time monitored access records and load state analysis, the high-load data blocks are recognized and migrated to the dynamic logic fragments for storage, and the overload problem of the basic logic fragments is relieved. Meanwhile, the low-load data blocks are moved back to the basic logic fragments, dynamic adjustment and balance of storage resources are further achieved, the load balancing capacity of the system is improved, and the requirements for high concurrency and real-time load changes in a financial scene are met.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and financial technology, and in particular to a database load optimization method, device, equipment and storage medium based on dynamic sharding. Background Art

[0002] As the financial industry's demand for data management and processing continues to grow, distributed databases are widely used in financial scenarios. However, existing sharding technologies have the following technical problems in dynamic load balancing and hot data processing, which seriously restrict their performance in scenarios such as high-frequency trading, real-time query, and large-scale report generation:

[0003] Existing sharding algorithms (such as sequential sharding and hash sharding) lack the ability to dynamically adjust data distribution and cannot respond in real time to changes in node load. When some logical shards are overloaded due to centralized access to hot data, existing solutions are difficult to achieve effective diversion at the sharding level, resulting in a performance bottleneck for a single node.

[0004] The identification and processing of hot data mostly remain at the business logic level, such as avoiding direct database access through caching or queue mechanisms. However, these methods fail to provide dynamic solutions at the database sharding management level, making it difficult to accurately and efficiently shard and migrate hot data.

[0005] In the prior art, sharding rules are usually set when the database is initialized, and it is difficult to dynamically adjust them according to changes in node load status or data access patterns. When the node load is unbalanced, the existing sharding method cannot effectively alleviate the load pressure by transferring high-load shards to other storage nodes through the dynamic sharding migration mechanism.

[0006] In summary, existing technologies have significant deficiencies in dynamic load balancing and hot data processing, and cannot meet the requirements of high concurrency, high flexibility and dynamic load adjustment in financial scenarios. Summary of the invention

[0007] The main purpose of the present invention is to provide a database load optimization method, device, equipment and storage medium based on dynamic sharding, aiming to solve the technical problem that the prior art lacks a dynamic sharding migration mechanism based on real-time load status in dynamic load balancing and hot data processing, and is difficult to effectively alleviate the uneven node load.

[0008] To achieve the above object, the present invention provides a database load optimization method based on dynamic sharding, comprising:

[0009] Configure storage nodes and basic logical shards, distribute data to basic logical shards based on data distribution strategies, and define dynamic logical shards as expansion units of the basic logical shards;

[0010] When establishing a database, the basic logical slice is associated with the storage node, and an association relationship between the dynamic logical slice, the basic logical slice and the storage node is established;

[0011] By monitoring the data access records of the basic logical shards, hotspot data whose access frequency exceeds an access threshold is identified, and the hotspot data is marked as a data block to be migrated;

[0012] When the load state of the basic logical shard exceeds a load threshold, a high-load data block is selected from the data blocks to be migrated according to a data migration rule, and the high-load data block is migrated to a storage unit of the dynamic logical shard;

[0013] When the access frequency of the data blocks stored in the dynamic logical shard drops below or equal to the access threshold, a low-load data block is selected from the stored data blocks according to the data migration rule, and the low-load data block is migrated from the dynamic logical shard back to the basic logical shard.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides a database load optimization device based on dynamic sharding, comprising:

[0015] A shard configuration module is used to configure storage nodes and basic logical shards, allocate data to basic logical shards based on data distribution strategies, and define dynamic logical shards as expansion units of the basic logical shards;

[0016] A database initialization module, used to associate the basic logical slice with the storage node when establishing a database, and to establish an association relationship between the dynamic logical slice and the basic logical slice and the storage node;

[0017] A data access monitoring module, used to identify hot data whose access frequency exceeds an access threshold by monitoring the data access records of the basic logical shards, and mark the hot data as data blocks to be migrated;

[0018] A data migration module, configured to select a high-load data block from the data blocks to be migrated according to a data migration rule and migrate the high-load data block to a storage unit of a dynamic logical slice when the load state of the basic logical slice exceeds a load threshold;

[0019] A data migration module is used to select low-load data blocks from the stored data blocks according to the data migration rules when the access frequency of the data blocks stored in the dynamic logical shard drops to less than or equal to the access threshold, and migrate the low-load data blocks from the dynamic logical shard back to the basic logical shard.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a database load optimization program based on dynamic sharding stored in the memory and executable on the processor, wherein the database load optimization program based on dynamic sharding, when executed by the processor, implements the steps of the database load optimization method based on dynamic sharding as described above.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a database load optimization program based on dynamic sharding is stored, and when the database load optimization program based on dynamic sharding is executed by a processor, the steps of the database load optimization method based on dynamic sharding as described above are implemented.

[0022] Beneficial effects: The present invention relates to the fields of big data technology and financial technology, and discloses a database load optimization method based on dynamic sharding. By configuring basic logical sharding and dynamic logical sharding, an association relationship between sharding and storage nodes is established; data access records are monitored, hot data is identified and marked as data blocks to be migrated; when the shard load exceeds the threshold, high-load data blocks are migrated to dynamic logical sharding; when the access frequency decreases, low-load data blocks are migrated back to the basic logical sharding. The present invention identifies and migrates high-load data blocks to dynamic logical sharding storage through the configuration and management of dynamic logical sharding, combined with real-time monitoring of access records and load status analysis, to alleviate the overload problem of basic logical sharding. At the same time, low-load data blocks are migrated back to the basic logical sharding, further realizing dynamic adjustment and balancing of storage resources, improving the load balancing capability of the system, and adapting to the needs of high concurrency and real-time load changes in financial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 A schematic diagram of an application environment of a database load optimization method based on dynamic sharding in an embodiment of the present invention;

[0025] Figure 2 It is a flow chart of an embodiment of a method for optimizing database load based on dynamic sharding according to the present invention;

[0026] Figure 3 It is a functional module diagram of a preferred embodiment of a database load optimization device based on dynamic sharding of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0028] Figure 5FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0030] The database load optimization method based on dynamic sharding provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can configure the basic logical sharding and dynamic logical sharding through the user end, establish an association relationship between the sharding and the storage node; monitor data access records, identify hot data and mark it as a data block to be migrated; when the sharding load exceeds the threshold, migrate the high-load data block to the dynamic logical sharding; when the access frequency decreases, migrate the low-load data block back to the basic logical sharding. The present invention identifies and migrates high-load data blocks to dynamic logical sharding storage through the configuration and management of dynamic logical sharding, combined with real-time monitoring of access records and load status analysis, to alleviate the overload problem of the basic logical sharding. At the same time, the low-load data blocks are migrated back to the basic logical sharding, further realizing the dynamic adjustment and balance of storage resources, improving the load balancing capability of the system, and adapting to the needs of high concurrency and real-time load changes in financial scenarios. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a database load optimization method based on dynamic sharding provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the database load optimization method based on dynamic sharding proposed by the present invention includes the following steps:

[0033] S10, configuring storage nodes and basic logical shards, allocating data to basic logical shards based on a data distribution strategy, and defining dynamic logical shards as expansion units of the basic logical shards;

[0034] In this embodiment, configuring storage nodes means allocating independent storage resources and identifiers to each physical storage unit in the database for storing and managing sharded data. Basic logical sharding means dividing data into multiple logical units according to specific rules and distributing them to different storage nodes to improve data access efficiency and load balancing capabilities.

[0035] Deploy multiple storage nodes and configure independent storage capacity, access control policy and unique identifier for each node; initialize multiple logical shards in the database management system, assign identifiers and set data storage paths; configure the mapping relationship between shards and nodes and record it in the metadata management module.

[0036] Data distribution strategy refers to allocating data to logical shards according to preset rules (such as hash sharding, range sharding, etc.) to ensure even distribution of data and avoid load concentration.

[0037] Define data distribution strategies, which can be based on data characteristics (such as access frequency, data size, etc.); use sharding rules to logically map input data and distribute data to specified basic logical shards; call distribution strategies when writing and querying data to ensure the effectiveness of sharding rules.

[0038] Dynamic logical sharding refers to a sharding unit that can be dynamically adjusted according to the load. It is an extension of the basic logical sharding and is used to store data blocks that exceed the load threshold to achieve load balancing and sharding capacity expansion.

[0039] Assign a unique identifier to each dynamic logical shard and initialize the storage path; configure the expansion rules of the dynamic logical shard, including trigger conditions (such as access frequency or storage capacity exceeding a threshold); record the association between the dynamic logical shard and the basic logical shard in the shard management module.

[0040] By expanding the storage capacity of basic logical shards through dynamic logical sharding and dynamically allocating data in combination with data distribution strategies, load balancing and storage optimization can be achieved. This can effectively avoid the problem of single shard overload or resource waste, improve the operating efficiency of distributed databases, and adapt to the needs of high-concurrency and dynamically changing financial services.

[0041] S20, when establishing a database, associating the basic logical shard with the storage node, and establishing an association relationship between the dynamic logical shard, the basic logical shard, and the storage node;

[0042] In this embodiment, during the database creation phase, each basic logical shard is mapped to a corresponding storage node through a metadata management module to form an association relationship between the shard and the node for storage and management of shard data.

[0043] Initialize the metadata management module and create a mapping table for the basic logical shards; record the identifier of each basic logical shard, the identifier of the associated storage node, the data storage path and access rights in the mapping table; load the mapping table during the database initialization process to implement the binding of the logical shards and storage nodes.

[0044] Through the extension mechanism, associated extended shards (dynamic logical shards) are dynamically created for the basic logical shards. The function of the dynamic logical shards is to share the storage pressure of the basic logical shards.

[0045] Assign a unique identifier to the dynamic logical shard; configure the expansion rules of the dynamic logical shard (such as trigger conditions, capacity restrictions); in the metadata management module, record the association between the basic logical shard and the dynamic logical shard, including its identifier and storage node location.

[0046] When a dynamic logical shard is created, it is bound to a specific storage node to ensure effective storage and access to the dynamic shard data.

[0047] When creating a dynamic logical shard, select a storage node with a lower load as its target node; add an association record between the dynamic logical shard and the storage node in the mapping table; configure the data access path of the dynamic shard to ensure that the data read and write requests of the dynamic shard can be correctly routed to the corresponding storage node.

[0048] By establishing the association between basic logical shards and storage nodes and dynamic logical shards, accurate storage and management of shard data is achieved, while providing a flexible expansion mechanism for dynamic load optimization. The binding of dynamic logical shards and storage nodes can efficiently share pressure when the load is too high, optimize system performance, and enhance the dynamic adaptability of distributed databases.

[0049] S30, identifying hotspot data whose access frequency exceeds an access threshold by monitoring the data access records of the basic logical shards, and marking the hotspot data as data blocks to be migrated;

[0050] In this embodiment, the log collection module monitors all data access records of the basic logical shards in real time, including the time, type, frequency and query conditions of the access. The purpose of this step is to obtain the data block access behavior in the shards and provide basic data for subsequent analysis.

[0051] Deploy the log collection module and set monitoring rules for the basic logical shards; the collected data includes the identifier of each data block, the access type (such as query or write), the access time and the query scope; the collected access records are stored in the log database in real time for subsequent analysis.

[0052] By using the collected data access records, the access frequency of each data block is counted to determine whether it exceeds the preset access threshold, thereby identifying the hot data blocks. The access threshold can be dynamically adjusted according to specific business needs.

[0053] Use the data analysis module to perform real-time statistics on the log database and calculate the access frequency of each data block; compare the access frequency of each data block with the access threshold; and mark the data blocks whose access frequency exceeds the threshold as hot data.

[0054] The identified hot data blocks are marked and their storage locations in the shards are recorded for easy migration in subsequent steps.

[0055] Generate a unique identifier for each hot data block; add the migration task of the data block to the task queue, including its storage location and priority information; update the metadata management module and mark the data block as "to be migrated".

[0056] By real-time monitoring of data access records of basic logical shards and identifying hot data based on access thresholds, hot data blocks can be efficiently distinguished and marked as objects to be migrated. This mechanism not only avoids overload of shards caused by hot data, but also provides data support for subsequent load balancing and dynamic migration, significantly improving the processing efficiency of the system.

[0057] S40, when the load state of the basic logical shard exceeds a load threshold, selecting a high-load data block from the data blocks to be migrated according to a data migration rule, and migrating the high-load data block to a storage unit of the dynamic logical shard;

[0058] In this embodiment, the load monitoring module monitors the load status of the basic logic slice in real time to determine whether it exceeds a preset load threshold. The load status may include multiple indicators, such as CPU usage, memory usage, IO access frequency, etc.

[0059] Deploy a load monitoring module to collect the load status of the shard in real time; set load thresholds, such as 80% CPU occupancy and 70% storage utilization; compare the real-time collected data with the threshold to determine whether the shard load is too high.

[0060] According to the data migration rules, high-load data blocks are selected from the data blocks marked to be migrated. The selection criteria of high-load data blocks can be comprehensively evaluated based on multi-dimensional indicators such as access frequency, data block size, and resource usage.

[0061] Data migration rules include: the access frequency exceeds a certain ratio, the data block size has a greater impact on storage resources, etc.

[0062] Use the filtering module to sort the data blocks to be migrated, give priority to the data blocks with high access frequency and large resource usage; mark the selected data blocks as "high-load data blocks".

[0063] Perform data block migration operations to transfer high-load data blocks from basic logical shards to dynamic logical shards. This step is intended to share the storage and computing pressure of basic logical shards.

[0064] Generate a migration task for a high-load data block, which includes a data block identifier, current storage location, target storage location, and migration priority; execute the migration task to move the data file of the data block to the storage path corresponding to the dynamic logical shard; update the metadata management module to record the new storage location of the data block and the load changes of the basic logical shard and the dynamic logical shard.

[0065] By monitoring the load status of the basic logical shards and selecting high-load data blocks to migrate to the dynamic logical shards according to the data migration rules, the load pressure of the basic logical shards can be effectively alleviated and dynamic load balancing can be achieved. The migration of data blocks and the synchronous update of metadata ensure the consistency of system data and the efficiency of operations.

[0066] S50, when the access frequency of the data blocks stored in the dynamic logical shard drops to less than or equal to the access threshold, select a low-load data block from the stored data blocks according to the data migration rule, and migrate the low-load data block from the dynamic logical shard back to the basic logical shard.

[0067] In this embodiment, the data blocks in the dynamic logical shards are monitored in real time by the access frequency monitoring module. The access frequency is a key indicator for measuring the activity of the data blocks. It is specifically defined as the number of times the data blocks are accessed per unit time. The access frequency needs to be compared with the preset access threshold to determine whether the low load standard is met.

[0068] An access frequency monitoring module is deployed on the storage nodes of the dynamic logical sharding, which regularly collects access information of each data block. The access information includes: the unique identifier of the data block; the timestamp of each access; and the access type (such as read or write operation).

[0069] The access frequency monitoring module is used to count the number of accesses to each data block in a unit of time. For example, if a data block is accessed 5 times in one minute, its access frequency is 5 times / minute.

[0070] Set an access threshold, for example, ≤ 3 accesses per minute. Compare the access frequencies of all data blocks with the threshold, and filter out data blocks with access frequencies lower than or equal to the threshold.

[0071] The screening of low-load data blocks is based on a comprehensive analysis of access frequency and other load indicators, such as the size of the data blocks and the occupancy rate of dynamic logical sharding resources.

[0072] In addition to access frequency, the following indicators are collected: the size of the data block; the consumption of storage resources by the data block, such as the I / O usage caused by read and write operations; and the query range size of the data block (indicating how active it is in being accessed).

[0073] Filter according to the indicator priority, for example: prioritize the data blocks with low access frequency; if the access frequencies are the same, further filter the data blocks that occupy a large storage space.

[0074] The filtered data blocks that meet the low-load condition are marked as "potential low-load data blocks" to prepare for the next migration operation.

[0075] A specific migration task is generated for each low-load data block, and the task content includes the data block identifier, current storage path, target storage path and migration task priority.

[0076] The system automatically generates migration tasks for the marked potential low-load data blocks; each task requires the following information:

[0077] Data block identifier (uniquely identifies each data block); current storage path (location of dynamic logical shard storage node); target storage path (location of basic logical shard storage node); migration task priority (high, medium, or low priority is determined based on the load level of the data block).

[0078] All tasks are added to the migration task queue, and the queue is sorted according to priority to ensure that high-priority tasks are executed first.

[0079] According to the priority in the task queue, the migration of low-load data blocks is performed in sequence, and the data blocks are moved from the dynamic logical shards to the basic logical shards.

[0080] The marked low-load data blocks are copied from the dynamic logical slice to the target basic logical slice. After the copy is completed, the original data block files in the dynamic logical slice are deleted to release storage resources.

[0081] Update the metadata management module to record the new storage location of the data block. At the same time, update the load status information of the dynamic logical shards and basic logical shards.

[0082] By detecting the changes in the access frequency of data blocks in the dynamic logical shards and screening the low-load data blocks to migrate back to the basic logical shards, the storage and computing resources of the dynamic logical shards can be released, and the flexibility of the dynamic logical shards in processing future hot data can be improved. At the same time, the storage load of the basic logical shards and the dynamic logical shards can be balanced, effectively improving the efficiency and stability of the overall system.

[0083] The present invention relates to the fields of big data technology and financial technology, and discloses a database load optimization method based on dynamic sharding. By configuring basic logical sharding and dynamic logical sharding, an association relationship between sharding and storage nodes is established; data access records are monitored, hot data is identified and marked as data blocks to be migrated; when the shard load exceeds a threshold, high-load data blocks are migrated to dynamic logical sharding; when the access frequency decreases, low-load data blocks are migrated back to basic logical sharding. The present invention identifies and migrates high-load data blocks to dynamic logical sharding storage through the configuration and management of dynamic logical sharding, combined with real-time monitoring of access records and load status analysis, to alleviate the overload problem of basic logical sharding. At the same time, low-load data blocks are migrated back to basic logical sharding, further realizing dynamic adjustment and balancing of storage resources, improving the load balancing capability of the system, and adapting to the needs of high concurrency and real-time load changes in financial scenarios.

[0084] In one embodiment, the above S10 includes:

[0085] S101, deploy multiple storage nodes, configure independent storage resources for each storage node and assign a unique identifier;

[0086] S102, initializing multiple basic logical shards on each storage node, assigning a unique identifier to each basic logical shard and setting a data storage path, access control policy and initial shard capacity;

[0087] S103, defining a data distribution strategy according to data access characteristics and distribution rules, and allocating data to the basic logical shards according to the data distribution strategy, wherein the data distribution strategy includes access frequency distribution, hash sharding or range sharding rules;

[0088] S104, configuring a dynamic logical slice as an expansion unit of the basic logical slice, and defining an expansion rule of the dynamic logical slice, wherein the expansion rule includes an expansion trigger condition and a capacity adjustment strategy.

[0089] In this embodiment, storage nodes are the basis of distributed storage of the database. Each node independently manages a portion of data. Independent storage resources ensure isolation between nodes. A unique identifier is used to mark and manage each storage node.

[0090] In a distributed system, multiple physical or virtual storage nodes are deployed. Nodes can be servers, containers, or cloud service instances. Each storage node is allocated independent storage resources, such as disk space, I / O bandwidth, and memory capacity. Each node is assigned a unique identifier, which is stored in the metadata management module and used to associate logical shards with storage nodes.

[0091] The basic logical shard is the basic unit of database distributed management. The unique identifier is used to accurately mark the shard. The storage path defines the location of the shard data. The access control policy manages the data access rights of the shard. The initial shard capacity sets the initial range of the shard storage data.

[0092] Create multiple logical shards on each storage node and assign unique identifiers, such as UUIDs, to the shards. Specify the data storage path for each shard, such as / data / partition1, which is stored in the metadata management module. Define access permission rules, such as read and write permissions based on user roles. Set the initial storage capacity of each logical shard, such as 10GB, to manage the boundaries of storage allocation.

[0093] The data distribution strategy maps data to corresponding logical shards based on data access frequency, distribution pattern and other characteristics to ensure that data is evenly distributed among shards.

[0094] Data distribution strategies can include:

[0095] Access frequency distribution: distributed to different shards according to the access frequency;

[0096] Hash sharding: Calculate the hash value based on the data key value and map the data to the shard;

[0097] Range sharding: Data is assigned to shards based on the data range (such as time period).

[0098] According to the above distribution rules, the input data is mapped to the corresponding basic logical shards. Example: A range sharding rule maps the data from 2024-01-01 to 2024-01-31 to shard 1.

[0099] Dynamic logical sharding is an extension of basic logical sharding, mainly used to relieve the load pressure of basic sharding. The expansion rule defines the conditions for triggering dynamic sharding expansion and the capacity adjustment strategy after expansion.

[0100] Assign dynamic logical shards to each basic logical shard and set expansion conditions, such as triggering when the load status reaches 80%. The expansion rules include:

[0101] Extended trigger conditions: such as access frequency exceeding the threshold;

[0102] Capacity adjustment strategy: Dynamic sharding expansion capacity is adjusted in 10GB increments.

[0103] This embodiment realizes distributed storage and balanced distribution of data through the configuration of storage nodes and basic logical shards, and provides dynamic expansion capabilities, so that the system can automatically adjust the storage configuration when the data load fluctuates, thereby improving the scalability and response efficiency of the system.

[0104] In one embodiment, the above S20 includes:

[0105] S201, creating and maintaining a mapping table between basic logical slices and storage nodes in a metadata management module, wherein the mapping table is used to record an identifier of each basic logical slice, an associated storage node identifier, a data storage path, and access rights;

[0106] S202, establishing an association relationship between each dynamic logical shard and a corresponding basic logical shard, wherein the association relationship includes an identifier of the dynamic logical shard, an identifier of the basic logical shard to which it belongs, and an associated storage node identifier;

[0107] S203, configuring a data access path for each dynamic logical shard, and recording the data access path in a metadata management module;

[0108] S204, during the database initialization process, load and verify the association relationship between all logical shards and storage nodes.

[0109] In this embodiment, establishing a database refers to the initialization phase of the database, which includes operations such as configuring sharding rules, registering and loading metadata, etc. This phase is an important part of system configuration, ensuring that the association between basic logical sharding, dynamic logical sharding, and storage nodes is correctly established, providing a basis for subsequent data allocation and dynamic sharding expansion.

[0110] During the database establishment phase, the user-defined database configuration file is loaded through the management module. The configuration file includes: the initial association between the basic logical shard and the storage node; the expansion strategy of the dynamic logical shard. When the database is started, the system loads the mapping information between the basic logical shard and the storage node from the metadata management module. Verify whether the association between the basic logical shard and the dynamic logical shard is correct, and ensure that the dynamic logical shard can be used as an expansion unit of the basic logical shard.

[0111] Based on user configuration, the initial storage path and resources are allocated to each dynamic logical slice, for example:

[0112] D_PART_01 is stored in NODE_C, the path is / data / node_c / d_part_01.

[0113] The expansion rules of dynamic logical sharding are recorded in the metadata management module, including expansion trigger conditions and capacity adjustment strategies.

[0114] Configure the storage paths of basic logical shards and dynamic logical shards so that the database service module can quickly access the corresponding shards through unique identifiers.

[0115] Verify the accessibility of the storage node, check the path validity and access rights. If there is an error, return the message of initialization failure and require reconfiguration.

[0116] The metadata management module is used to maintain the core configuration information of the database logical shards. The mapping table provides the relationship between the basic logical shards and the storage nodes, including unique identification, storage location, and permission control information.

[0117] Initialize the metadata management module and create a mapping table for basic logical shards. The data fields include:

[0118] The basic logical shard identifier, such as PART_01; the storage node identifier, such as NODE_A; the data storage path, such as / data / node_a / part_01; and the access permission, such as read-only or read-write permission.

[0119] Dynamically update the mapping table to reflect the migration of storage nodes or path changes of the shards. Ensure the accuracy of the mapping relationship through regular verification.

[0120] The association relationship of dynamic logical shards is used to clarify their subordinate relationships and storage locations, ensuring that dynamic logical shards can be quickly located and operated when expanding storage or migrating loads.

[0121] Assign a unique identifier to the dynamic shard, such as D_PART_01. Specify the corresponding underlying logical shard identifier, such as PART_01. Record the storage node associated with the dynamic shard, such as NODE_B.

[0122] The association relationship is stored in the metadata management module, and an interface is provided for fast update and query during dynamic sharding expansion.

[0123] The data access path is the physical storage address of the dynamic logical slice. Recording the path information facilitates data access and management. Specify the storage path of the dynamic logical slice, for example, / data / node_b / d_part_01. Ensure that the path is unique and conforms to the directory structure of the storage node.

[0124] Add a new field in the mapping table of the metadata management module to record the access path of the dynamic shard. Support dynamic update of path information through the management interface.

[0125] When the database is initialized, the association between all shards and storage nodes is loaded and verified to ensure that the shards can be accessed normally and meet the configuration requirements.

[0126] Load the mapping table between shards and storage nodes from the metadata management module, and load the storage configuration of each shard according to the shard identifier.

[0127] Check whether the storage path is valid and ensure that the shard can access the corresponding storage location. Verify whether the access permissions are consistent with the actual shard configuration.

[0128] This embodiment records the association between the basic logical shards and the storage nodes in the metadata management module and establishes the association between the dynamic logical shards, thereby realizing efficient management between the logical shards and the physical storage. When the dynamic shards are expanded, the target storage nodes can be found quickly, which reduces the complexity of data migration and storage path configuration.

[0129] In one embodiment, the above S30 includes:

[0130] S301, deploying a log collection module, and monitoring data access records of the basic logical shards through the log collection module, wherein the data access records include a data block identifier, access time, access type, access frequency, and query range;

[0131] S302, based on the monitored data access records, filter data blocks whose access frequency exceeds an access threshold;

[0132] S303, analyzing the change trend of the access frequency of the screened data blocks, and determining whether the duration of the access frequency of the data blocks exceeding the access threshold satisfies the determination condition of the hotspot data;

[0133] S304: Mark the data blocks determined to be hot data as data blocks to be migrated, and add the data blocks to be migrated to a migration task queue.

[0134] In this embodiment, the deployment log collection module is a component designed to monitor the data access behavior of the basic logical shards, and is responsible for recording various indicators related to data access, including data block identifiers, access time, access type, access frequency, and query range.

[0135] Data block identifier: used to uniquely identify a data block in the database.

[0136] Access time: The timestamp of each data access, used to analyze access frequency and distribution.

[0137] Access type: distinguish between query, update, delete and other operation types.

[0138] Access frequency: counts the number of times a data block is accessed within a specified time period.

[0139] Query scope: records the data scope involved in the query conditions, reflecting the aggregation of data access.

[0140] Deploy the log collection module on the database service node to intercept and record data access requests in real time. Store log data in batches into a temporary log buffer and write it periodically to the distributed log storage system. Use streaming data processing tools (such as Flink and Kafka Streams) to analyze and store log data in real time.

[0141] Based on the collected access logs, the access frequency of each data block is counted, and the data blocks whose access frequency exceeds the set threshold are screened out as candidates for potential hot data.

[0142] Access threshold: A predefined access frequency standard used to distinguish frequently accessed data blocks.

[0143] Filtering logic: accumulate and count the number of accesses to each data block and compare its access frequency with the threshold.

[0144] Perform batch statistics on stored log data through a distributed computing engine (such as Spark or Hadoop). Use SQL queries to aggregate the access frequency of data blocks and output the screening results to the hotspot candidate data set.

[0145] The filtered high-frequency data blocks are further analyzed to evaluate whether their access frequency continues to exceed the threshold to determine whether they are true hot data.

[0146] Change trend: Evaluate the change curve of data block access frequency based on time series analysis.

[0147] Duration: Set judgment conditions to ensure that the data block maintains a high access frequency over a period of time.

[0148] The access logs of the filtered high-frequency data blocks are grouped and counted by time period. The average access frequency and growth rate of the data blocks are calculated using the sliding window technology to determine whether the high-frequency state duration of each data block exceeds the preset value.

[0149] Data blocks identified as hot spots need to be marked so that they can be included in the subsequent migration task management.

[0150] Annotation information: includes data block identifier, storage location, and hotspot marking time.

[0151] Migration task queue: provides management support for subsequent high-load data migration.

[0152] The information of the data blocks to be migrated is recorded in the metadata management module, and the marked data blocks to be migrated are added to the migration task queue according to the priority.

[0153] This embodiment can accurately identify real hot data by monitoring the data access records of the basic logical shards and dynamically analyzing the access frequency change trend. The hot data is marked as the data block to be migrated and added to the migration task queue, providing a clear data basis for the subsequent dynamic sharding load optimization operation. This not only improves the accuracy of data migration, but also avoids unnecessary migration operations, thereby optimizing system performance and resource utilization.

[0154] In one embodiment, the above S40 includes:

[0155] S401, deploying a load monitoring module, and monitoring the load status of the basic logic shard through the load monitoring module, wherein the load status includes multiple load indicators;

[0156] S402, when it is monitored that any load indicator of the basic logical shard exceeds the load threshold, a high-load data block is selected from the data blocks to be migrated according to the data migration rule;

[0157] S403, generating a migration task for the high-load data block, wherein the migration task includes a data block identifier, a current storage location, a storage location of a target dynamic logical slice, and migration priority information;

[0158] S404, executing the migration task according to the migration task queue to migrate the high-load data block from the basic logical shard to the storage unit of the dynamic logical shard;

[0159] S405, updating the metadata management module to record the new storage location of the high-load data block and the load status change of the basic logical shard.

[0160] In this embodiment, the core function of the load monitoring module is to monitor the operating status of the basic logical shards in real time, including multi-dimensional load indicators such as storage resources, computing resources and IO operations.

[0161] Storage resources: used to determine whether the storage capacity of the shard is close to the limit.

[0162] Computing resources: Evaluate computing load through CPU utilization.

[0163] IO operations: Statistics on the frequency of read and write operations, reflecting the access pressure of the shard.

[0164] A load monitoring module is deployed on each storage node to periodically collect the load status data of the shards and transmit it to the monitoring center for aggregation processing. For indicators that are updated frequently (such as IO operations), streaming analysis methods are used for real-time calculation.

[0165] When the load monitoring module reports that a certain load indicator exceeds a preset threshold, it is necessary to identify the specific data block that caused the overload. The basis for selecting the high-load data block includes:

[0166] Access frequency: Filter data blocks with significantly higher access volume than the average.

[0167] Resource consumption: Analyze the storage and computing resource consumption of each data block.

[0168] Query range concentration: determines whether the data block frequently appears in the query conditions.

[0169] Based on the indicators provided by the load monitoring module and combined with the access logs of the data blocks, the data blocks are screened according to the set rules, and the data blocks that meet the conditions are marked as high-load data blocks.

[0170] A migration task is a structured instruction that guides the migration of data from the current shard to the target dynamic logical shard. A migration task must contain the following key information:

[0171] Data block identifier: Identifies the data block that needs to be migrated.

[0172] Current storage location: points to the source shard of the data block.

[0173] Target dynamic logical shard location: used to record the storage shard after the data block is migrated.

[0174] Priority information: used to schedule the order of migration tasks.

[0175] The migration tasks are recorded in the metadata management module and added to the migration queue according to the priority. When the task is generated, the availability of the storage node needs to be checked to ensure that there will be no new overload after the migration.

[0176] The core purpose of executing the migration task is to transfer the high-load data blocks from the basic logical shard to the dynamic logical shard without interrupting the service, so as to alleviate the overload state of the current shard. The migration task is executed step by step according to the priority in the migration queue. After completion, the metadata record in the target dynamic logical shard is updated to ensure that the access request can be located at the new location.

[0177] After the migration is completed, the metadata management module needs to synchronously update the storage location of the data block and the load status of the shard to ensure that subsequent access can locate the correct location. The metadata management module updates the storage location table and the shard load status table to store the new location of each data block and the latest load data of the shard.

[0178] Through the above steps, this embodiment can dynamically identify and migrate high-load data blocks when the load status of the basic logical shard exceeds the threshold, thereby reducing the pressure on the basic logical shard; the dynamic logical shard improves the load balancing capability of the storage system by receiving migration tasks, ensures the efficient and stable operation of the distributed database, and provides flexibility for future expansion.

[0179] In one embodiment, the above S50 includes:

[0180] S501, deploying an access frequency monitoring module, and collecting access frequencies of all data blocks in the dynamic logical shards through the access frequency monitoring module;

[0181] S502, comparing the access frequency of each data block with the access threshold, and marking the data blocks whose access frequency is lower than or equal to the access threshold as potential low-load data blocks;

[0182] S503, selecting a low-load data block from the potential low-load data blocks according to the data migration rule;

[0183] S504, generating a migration task for the low-load data block, the migration task including a data block identifier, a current storage location, a storage path of a target basic logical slice, and a migration task priority;

[0184] S505: Migrate the low-load data block from the dynamic logical shard back to the basic logical shard according to the migration task.

[0185] In this embodiment, the access frequency monitoring module is used to dynamically monitor the access status of each data block in the shard, and mainly records the following information: data block identifier, which is used to uniquely identify each data block; access frequency, that is, the number of accesses per unit time.

[0186] Deploy the access frequency monitoring module on the storage node of the dynamic logical sharding, regularly collect the access log of each data block, and store the access frequency in the monitoring database for subsequent analysis.

[0187] Comparison of access frequency with access threshold is the core step to determine whether a data block is a potential low-load data block: data blocks below or equal to the threshold are marked as potential low-load data blocks; the comparison operation needs to ensure that the access frequency data has a consistent time period with the threshold.

[0188] The monitoring module is used to filter the collected access frequency data and select the data blocks that meet the conditions. The marked results are stored in a queue to be processed for subsequent selection.

[0189] Data migration rules are standards for ensuring performance during dynamic logical shard migration. Consider the following when selecting low-load data blocks:

[0190] The size of the data block (small data blocks are preferred); the distribution density of the data blocks (even distribution is preferred); the access interval of the data blocks (longer intervals are preferred).

[0191] Data blocks marked as potentially low-load are scored and sorted according to the migration rules, with the highest-scoring data blocks being selected first.

[0192] A migration task is a set of instructions that contains all the information required for the migration operation:

[0193] A data block identifier, used to identify the data block to be migrated;

[0194] The current storage location refers to the specific storage path of the data block in the dynamic logical shard;

[0195] The target storage path, which points to the new location in the underlying logical shard;

[0196] The priority of the migration task is used to schedule the execution order of tasks.

[0197] Generate a migration task in the metadata management module, add the task to the task queue, and allocate execution resources to the task.

[0198] The goal of data migration is to migrate the data blocks marked as low load from the dynamic logical shards back to the basic logical shards to optimize the load distribution. Traverse the task queue and perform the migration operations in sequence according to the priority of the tasks. After the migration is completed, update the metadata to record the new storage location and the status of the basic logical shards.

[0199] Through the above steps, when the access frequency of data blocks in the dynamic logical shard decreases, the low-load data blocks can be identified and relocated in time, thereby releasing the storage resources of the dynamic logical shard and restoring the balanced load state of the basic logical shard. This effectively improves the utilization rate of storage nodes, reduces the resource waste of dynamic logical shards, and ensures the efficient operation and resource optimization of the database.

[0200] In one embodiment, after the above S50, the method further includes:

[0201] S601, monitoring the current load status of the storage node and the basic logical slice, wherein the current load status includes CPU occupancy, storage usage, and data access frequency;

[0202] S602, analyzing the current load state to generate current load distribution of dynamic logical slices and basic logical slices;

[0203] S603, based on the current load distribution, analyzing whether the judgment criteria of the high-load data block and the migration conditions of the low-load data block in the current data migration rule meet the current load distribution, and generating an analysis result;

[0204] S604, optimizing the current data migration rule based on the analysis result, wherein the optimization includes adjusting the judgment criteria of the high-load data block or adjusting the migration condition of the low-load data block, and setting a migration priority for the low-load data block that meets the migration condition;

[0205] S605: Adjust the storage configuration of the dynamic logical shard based on the optimized data migration rule.

[0206] In this embodiment, the current load status is an important indicator reflecting the operation status of the storage node and the basic logic shard, including:

[0207] CPU usage: evaluates the computing resource usage caused by sharding operations;

[0208] Storage usage: Analyze storage capacity occupancy and available space;

[0209] Data access frequency: reflects the read and write activity of data blocks in the shard.

[0210] Deploy the monitoring module to regularly collect the load indicators of storage nodes and basic logical shards, and store the collected data in the load monitoring database for subsequent analysis.

[0211] The current load distribution is an overall assessment of the load status of multiple shards. The purpose is to identify imbalances: by comparing the load status of dynamic logical shards and basic logical shards, determine whether the load distribution needs to be optimized; analyze the relative load between shards, such as CPU usage differences, storage usage differences, etc.

[0212] Data aggregation and normalization are applied to the collected load indicators to generate a load distribution diagram for each shard, which is then stored in the metadata management module for subsequent optimization.

[0213] Data migration rules are the core rules for dynamic logical sharding and must match the current load distribution:

[0214] Criteria for determining high-load data blocks: for example, the access frequency is higher than the average multiple;

[0215] Low-load data block migration conditions: for example, the access frequency continues to be lower than a certain threshold.

[0216] Compare the current migration rules with the actual load distribution, identify incompatible items in the rules, and generate analysis results including adjustment suggestions.

[0217] The optimization process aims to improve the adaptability of data migration rules, including:

[0218] Adjust the criteria for determining high-load data blocks, such as lowering the determination threshold;

[0219] Adjust the conditions for repatriating low-load data blocks, such as increasing the time determination for access frequency.

[0220] Modify the rule parameters based on the analysis results and update the optimized rules in the metadata management module.

[0221] The storage configuration of dynamic logical sharding should adapt to the optimized rules, including:

[0222] Expand the storage capacity of dynamic shards;

[0223] Adjust the relationship between dynamic logical shards and basic logical shards.

[0224] Implement dynamic sharding adjustments through configuration management tools, record new storage configurations in the metadata management module, and ensure the efficiency of migration rule execution.

[0225] Through the above steps, this embodiment can monitor the load status of storage nodes and logical shards in real time, and dynamically adjust data migration rules to adapt to the actual load distribution. At the same time, the storage configuration of dynamic logical shards is optimized to effectively solve the problem of uneven shard load, thereby improving system performance, increasing the utilization rate of storage nodes, and ensuring the stability and efficiency of database operation.

[0226] In one embodiment, a database load optimization device based on dynamic sharding is provided, and the database load optimization device based on dynamic sharding corresponds to the database load optimization method based on dynamic sharding in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a database load optimization device based on dynamic sharding of the present invention. Sharding configuration module 10, database initialization module 20, data access monitoring module 30, data migration module 40 and data relocation module 50. The functional modules are described in detail as follows:

[0227] A shard configuration module 10 is used to configure storage nodes and basic logical shards, allocate data to basic logical shards based on a data distribution strategy, and define dynamic logical shards as expansion units of the basic logical shards;

[0228] A database initialization module 20 is used to associate the basic logical slice with the storage node when establishing a database, and to establish an association relationship between the dynamic logical slice and the basic logical slice and the storage node;

[0229] A data access monitoring module 30, configured to identify hotspot data whose access frequency exceeds an access threshold by monitoring data access records of the basic logical shards, and identify the hotspot data as data blocks to be migrated;

[0230] A data migration module 40, configured to select a high-load data block from the data blocks to be migrated according to a data migration rule and migrate the high-load data block to a storage unit of a dynamic logical slice when the load state of the basic logical slice exceeds a load threshold;

[0231] The data migration module 50 is used to select low-load data blocks from the stored data blocks according to the data migration rules when the access frequency of the data blocks stored in the dynamic logical shard drops to less than or equal to the access threshold, and migrate the low-load data blocks from the dynamic logical shard back to the basic logical shard.

[0232] In one embodiment, the shard configuration module 10 is specifically configured to:

[0233] Deploy multiple storage nodes, configure independent storage resources for each storage node and assign a unique identifier;

[0234] Initialize multiple basic logical shards on each storage node, assign a unique identifier to each basic logical shard, and set the data storage path, access control policy, and initial shard capacity;

[0235] According to the data access characteristics and distribution rules, a data distribution strategy is defined, and the data is distributed to the basic logical shards according to the data distribution strategy, wherein the data distribution strategy includes access frequency distribution, hash sharding or range sharding rules;

[0236] A dynamic logical shard is configured as an expansion unit of the basic logical shard, and an expansion rule of the dynamic logical shard is defined, wherein the expansion rule includes an expansion trigger condition and a capacity adjustment strategy.

[0237] In one embodiment, the database initialization module 20 is specifically used to:

[0238] Create and maintain a mapping table between basic logical slices and storage nodes in the metadata management module, wherein the mapping table is used to record the identifier of each basic logical slice, the associated storage node identifier, the data storage path and the access rights;

[0239] Establishing an association relationship between each dynamic logical shard and a corresponding basic logical shard, wherein the association relationship includes an identifier of the dynamic logical shard, an identifier of the basic logical shard to which it belongs, and an associated storage node identifier;

[0240] Configure the data access path of each dynamic logical shard, and record the data access path in the metadata management module;

[0241] During database initialization, the association between all logical shards and storage nodes is loaded and verified.

[0242] In one embodiment, the data access monitoring module 30 is specifically used to:

[0243] Deploy a log collection module, and monitor the data access records of the basic logical shards through the log collection module, wherein the data access records include a data block identifier, access time, access type, access frequency, and query range;

[0244] Based on the monitored data access records, filter the data blocks whose access frequency exceeds the access threshold;

[0245] Analyze the changing trend of the access frequency of the selected data blocks, and determine whether the duration of the access frequency of the data blocks exceeding the access threshold meets the determination condition of the hot data;

[0246] The data blocks determined to be hot data are marked as data blocks to be migrated, and the data blocks to be migrated are added to a migration task queue.

[0247] In one embodiment, the data migration module 40 is specifically configured to:

[0248] Deploy a load monitoring module, and monitor the load status of the basic logical shard through the load monitoring module, wherein the load status includes multiple load indicators;

[0249] When any load indicator of the basic logical shard is monitored to exceed the load threshold, a high-load data block is selected from the data blocks to be migrated according to the data migration rule;

[0250] Generate a migration task for the high-load data block, the migration task including a data block identifier, a current storage location, a storage location of a target dynamic logical slice, and migration priority information;

[0251] Execute the migration task according to the migration task queue to migrate the high-load data block from the basic logical slice to the storage unit of the dynamic logical slice;

[0252] Update the metadata management module to record the new storage location of high-load data blocks and the load status changes of the basic logical shards.

[0253] In one embodiment, the data migration module 50 is specifically used for:

[0254] Deploy an access frequency monitoring module, and collect the access frequency of all data blocks in the dynamic logical shard through the access frequency monitoring module;

[0255] Comparing the access frequency of each data block with the access threshold, and marking the data blocks whose access frequency is lower than or equal to the access threshold as potential low-load data blocks;

[0256] selecting a low-load data block from the potential low-load data blocks according to the data migration rule;

[0257] Generate a migration task for the low-load data block, the migration task including a data block identifier, a current storage location, a storage path of a target basic logical slice, and a migration task priority;

[0258] The low-load data block is migrated from the dynamic logical slice back to the basic logical slice according to the migration task.

[0259] In one embodiment, the data migration module 50 is specifically used for:

[0260] Monitor the current load status of the storage nodes and basic logical slices, the current load status including CPU occupancy, storage usage and data access frequency;

[0261] Analyze the current load state to generate current load distribution of dynamic logical slices and basic logical slices;

[0262] Based on the current load distribution, analyzing whether the judgment criteria of the high-load data block and the migration conditions of the low-load data block in the current data migration rule meet the current load distribution, and generating an analysis result;

[0263] Optimizing the current data migration rule based on the analysis result, wherein the optimization includes adjusting the judgment criteria of the high-load data block or adjusting the migration condition of the low-load data block, and setting a migration priority for the low-load data block that meets the migration condition;

[0264] Based on the optimized data migration rule, the storage configuration of the dynamic logical shard is adjusted.

[0265] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side method for database load optimization based on dynamic sharding.

[0266] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a database load optimization method based on dynamic sharding

[0267] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0268] Configure storage nodes and basic logical shards, distribute data to basic logical shards based on data distribution strategies, and define dynamic logical shards as expansion units of the basic logical shards;

[0269] When establishing a database, the basic logical slice is associated with the storage node, and an association relationship between the dynamic logical slice, the basic logical slice and the storage node is established;

[0270] By monitoring the data access records of the basic logical shards, hotspot data whose access frequency exceeds an access threshold is identified, and the hotspot data is marked as a data block to be migrated;

[0271] When the load state of the basic logical shard exceeds a load threshold, a high-load data block is selected from the data blocks to be migrated according to a data migration rule, and the high-load data block is migrated to a storage unit of the dynamic logical shard;

[0272] When the access frequency of the data blocks stored in the dynamic logical shard drops below or equal to the access threshold, a low-load data block is selected from the stored data blocks according to the data migration rule, and the low-load data block is migrated from the dynamic logical shard back to the basic logical shard.

[0273] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0274] Configure storage nodes and basic logical shards, distribute data to basic logical shards based on data distribution strategies, and define dynamic logical shards as expansion units of the basic logical shards;

[0275] When establishing a database, the basic logical slice is associated with the storage node, and an association relationship between the dynamic logical slice, the basic logical slice and the storage node is established;

[0276] By monitoring the data access records of the basic logical shards, hotspot data whose access frequency exceeds an access threshold is identified, and the hotspot data is marked as a data block to be migrated;

[0277] When the load state of the basic logical shard exceeds a load threshold, a high-load data block is selected from the data blocks to be migrated according to a data migration rule, and the high-load data block is migrated to a storage unit of the dynamic logical shard;

[0278] When the access frequency of the data blocks stored in the dynamic logical shard drops below or equal to the access threshold, a low-load data block is selected from the stored data blocks according to the data migration rule, and the low-load data block is migrated from the dynamic logical shard back to the basic logical shard.

[0279] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0280] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0281] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0282] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A database load optimization method based on dynamic sharding, characterized in that: The following steps are involved: Configure storage nodes and basic logical shards, distribute data to basic logical shards based on data distribution strategies, and define dynamic logical shards as expansion units of the basic logical shards; When establishing a database, the basic logical slice is associated with the storage node, and an association relationship between the dynamic logical slice, the basic logical slice and the storage node is established; By monitoring the data access records of the basic logical shards, hotspot data whose access frequency exceeds an access threshold is identified, and the hotspot data is marked as a data block to be migrated; When the load state of the basic logical shard exceeds a load threshold, a high-load data block is selected from the data blocks to be migrated according to a data migration rule, and the high-load data block is migrated to a storage unit of the dynamic logical shard; When the access frequency of the data blocks stored in the dynamic logical shard drops below or equal to the access threshold, a low-load data block is selected from the stored data blocks according to the data migration rule, and the low-load data block is migrated from the dynamic logical shard back to the basic logical shard.

2. The database load optimization method based on dynamic sharding according to claim 1, characterized in that: Configuring storage nodes and basic logical shards, allocating data to basic logical shards based on data distribution strategies, and defining dynamic logical shards as expansion units of the basic logical shards, including: Deploy multiple storage nodes, configure independent storage resources for each storage node and assign a unique identifier; Initialize multiple basic logical shards on each storage node, assign a unique identifier to each basic logical shard, and set the data storage path, access control policy, and initial shard capacity; According to the data access characteristics and distribution rules, a data distribution strategy is defined, and the data is distributed to the basic logical shards according to the data distribution strategy, wherein the data distribution strategy includes access frequency distribution, hash sharding or range sharding rules; A dynamic logical shard is configured as an expansion unit of the basic logical shard, and an expansion rule of the dynamic logical shard is defined, wherein the expansion rule includes an expansion trigger condition and a capacity adjustment strategy.

3. The database load optimization method based on dynamic sharding according to claim 1, characterized in that: When establishing a database, the basic logical slice is associated with the storage node, and an association relationship between the dynamic logical slice, the basic logical slice and the storage node is established, including: Create and maintain a mapping table between basic logical slices and storage nodes in the metadata management module, wherein the mapping table is used to record the identifier of each basic logical slice, the associated storage node identifier, the data storage path and the access rights; Establishing an association relationship between each dynamic logical shard and a corresponding basic logical shard, wherein the association relationship includes an identifier of the dynamic logical shard, an identifier of the basic logical shard to which it belongs, and an associated storage node identifier; Configure the data access path of each dynamic logical shard, and record the data access path in the metadata management module; During database initialization, the association between all logical shards and storage nodes is loaded and verified.

4. The database load optimization method based on dynamic sharding according to claim 1, characterized in that: By monitoring the data access records of the basic logical shards, identifying hotspot data whose access frequency exceeds an access threshold, and marking the hotspot data as data blocks to be migrated, including: Deploy a log collection module, and monitor the data access records of the basic logical shards through the log collection module, wherein the data access records include a data block identifier, access time, access type, access frequency, and query range; Based on the monitored data access records, filter the data blocks whose access frequency exceeds the access threshold; Analyze the changing trend of the access frequency of the selected data blocks, and determine whether the duration of the access frequency of the data blocks exceeding the access threshold meets the determination condition of the hot data; The data blocks determined to be hot data are marked as data blocks to be migrated, and the data blocks to be migrated are added to a migration task queue.

5. The database load optimization method based on dynamic sharding according to claim 1, characterized in that: When the load state of the basic logical shard exceeds a load threshold, a high-load data block is selected from the data blocks to be migrated according to a data migration rule, and the high-load data block is migrated to a storage unit of the dynamic logical shard, including: Deploy a load monitoring module, and monitor the load status of the basic logical shard through the load monitoring module, wherein the load status includes multiple load indicators; When any load indicator of the basic logical shard is monitored to exceed the load threshold, a high-load data block is selected from the data blocks to be migrated according to the data migration rule; Generate a migration task for the high-load data block, the migration task including a data block identifier, a current storage location, a storage location of a target dynamic logical slice, and migration priority information; Execute the migration task according to the migration task queue to migrate the high-load data block from the basic logical slice to the storage unit of the dynamic logical slice; Update the metadata management module to record the new storage location of high-load data blocks and the load status changes of the basic logical shards.

6. The database load optimization method based on dynamic sharding as described in claim 1, characterized in that: When the access frequency of the data blocks stored in the dynamic logical shard drops to less than or equal to the access threshold, selecting a low-load data block from the stored data blocks according to the data migration rule, and migrating the low-load data block from the dynamic logical shard back to the basic logical shard, including: Deploy an access frequency monitoring module, and collect the access frequency of all data blocks in the dynamic logical shard through the access frequency monitoring module; Comparing the access frequency of each data block with the access threshold, and marking the data blocks whose access frequency is lower than or equal to the access threshold as potential low-load data blocks; selecting a low-load data block from the potential low-load data blocks according to the data migration rule; Generate a migration task for the low-load data block, the migration task including a data block identifier, a current storage location, a storage path of a target basic logical slice, and a migration task priority; The low-load data block is migrated from the dynamic logical slice back to the basic logical slice according to the migration task.

7. The database load optimization method based on dynamic sharding according to claim 1, characterized in that: When the access frequency of the data blocks stored in the dynamic logical shard drops to less than or equal to the access threshold, a low-load data block is selected from the stored data blocks according to the data migration rule, and the low-load data block is migrated from the dynamic logical shard back to the basic logical shard, the method further includes: Monitor the current load status of the storage nodes and basic logical slices, the current load status including CPU occupancy, storage usage and data access frequency; Analyze the current load state to generate current load distribution of dynamic logical slices and basic logical slices; Based on the current load distribution, analyzing whether the judgment criteria of the high-load data block and the migration conditions of the low-load data block in the current data migration rule meet the current load distribution, and generating an analysis result; Optimizing the current data migration rule based on the analysis result, wherein the optimization includes adjusting the judgment criteria of the high-load data block or adjusting the migration condition of the low-load data block, and setting a migration priority for the low-load data block that meets the migration condition; Based on the optimized data migration rule, the storage configuration of the dynamic logical shard is adjusted.

8. A database load optimization device based on dynamic sharding, characterized in that: The database load optimization device based on dynamic sharding includes: A shard configuration module is used to configure storage nodes and basic logical shards, allocate data to basic logical shards based on data distribution strategies, and define dynamic logical shards as expansion units of the basic logical shards; A database initialization module, used to associate the basic logical slice with the storage node when establishing a database, and to establish an association relationship between the dynamic logical slice and the basic logical slice and the storage node; A data access monitoring module, used to identify hot data whose access frequency exceeds an access threshold by monitoring the data access records of the basic logical shards, and mark the hot data as data blocks to be migrated; A data migration module, configured to select a high-load data block from the data blocks to be migrated according to a data migration rule and migrate the high-load data block to a storage unit of a dynamic logical slice when the load state of the basic logical slice exceeds a load threshold; A data migration module is used to select low-load data blocks from the stored data blocks according to the data migration rules when the access frequency of the data blocks stored in the dynamic logical shard drops to less than or equal to the access threshold, and migrate the low-load data blocks from the dynamic logical shard back to the basic logical shard.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a dynamic sharding-based database load optimization program stored in the memory and executable on the processor. When the dynamic sharding-based database load optimization program is executed by the processor, the steps of the dynamic sharding-based database load optimization method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a database load optimization program based on dynamic sharding, and when the database load optimization program based on dynamic sharding is executed by the processor, the steps of the database load optimization method based on dynamic sharding as described in any one of claims 1-7 are implemented.

Images