Database cutting and storing method and system based on single data source

Abstract

The invention discloses a database cutting and storing method and system based on a single data source, and the method comprises the following steps: automatically generating an optimal partitioning strategy through a machine learning algorithm according to the characteristics and query mode of the single data source; according to the load condition of the system, the partition distribution is automatically adjusted, and the data volume and load balance among the partitions are ensured; a distributed index mechanism and a cache mechanism are introduced, and the performance of cross-partition query is optimized; the number of partitions is dynamically adjusted according to service requirements, and data is automatically migrated to adapt to a new partition strategy to achieve elastic expansion; and the system dynamically adjusts and stores the received single data source through an intelligent storage strategy. According to the database cutting and storing method and system based on the single data source, the query performance is improved, the number of partitions is dynamically adjusted according to service requirements, the system expansibility is enhanced, manual intervention is reduced, the maintenance cost is reduced, and the method and system are suitable for large-scale data storage and query scenes.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a database cutting and storage method and system based on a single data source. Background Art

[0002] With the rapid development of cloud computing and big data technologies, database systems are required to process ever-increasing amounts of data. Traditional single-table storage methods can no longer meet performance and scalability requirements. Consequently, database partitioning technology has emerged, improving query performance and management efficiency by dividing data into multiple partitions. However, existing technologies have the following shortcomings: (1) Partition granularity is too coarse: It cannot flexibly adapt to the needs of different business scenarios, resulting in poor query performance.

[0003] (2) Low cross-partition query efficiency: Data access and query performance between multiple partitions is poor, affecting the response time of the overall system.

[0004] (3) Difficulty in dynamic expansion: It is difficult to dynamically adjust the partitioning strategy according to business growth, resulting in poor system scalability.

[0005] (4) Low storage efficiency: Existing storage methods fail to fully utilize storage resources, resulting in a large amount of storage space being occupied and increasing storage costs.

[0006] (5) Insufficient data redundancy: There is a lack of effective data redundancy mechanism, and single point failure may lead to data loss.

[0007] (6) Insufficient data compression: There is no effective data compression mechanism, and a large amount of data takes up storage space, affecting storage efficiency.

[0008] (7) Insufficient storage engine optimization: Existing storage engines are not optimized for large-scale data, resulting in poor read and write performance.

[0009] Based on the above technical problems, the present invention proposes a database cutting and storage method and system based on a single data source, aiming to improve the performance, scalability and management efficiency of the database. Summary of the Invention

[0010] The purpose of the present invention is to provide a database segmentation and storage method and system based on a single data source, which significantly improves query performance by optimizing cross-partition queries and introducing distributed indexes; supports dynamic adjustment of the number of partitions according to business needs, significantly enhancing system scalability; and reduces the need for manual intervention through automated partition strategy generation and dynamic load balancing, thereby reducing maintenance costs, and is suitable for large-scale data storage and query scenarios.

[0011] To achieve the above objectives, the present invention provides a database segmentation and storage method based on a single data source, comprising the following steps: Step S1: Automatically generate the optimal partitioning strategy based on the characteristics and query pattern of a single data source through a machine learning algorithm; Step S2: Automatically adjust the partition distribution according to the system load to ensure data volume and load balance among the partitions; Step S3: Introduce a distributed indexing mechanism and a caching mechanism to optimize the performance of cross-partition queries; Step S4: Dynamically adjust the number of partitions according to business needs to achieve elastic expansion.

[0012] Preferably, in step S1, the characteristics of the single data source include: time range, value range, and geographical location; The intelligent partitioning strategy supports range partitioning, list partitioning, hash partitioning, and composite partitioning.

[0013] Preferably, in step S1, the specific process of generating the partition strategy is as follows: Step S11, data characteristic analysis: analyzing the distribution characteristics of a single data source; Step S12, historical query pattern analysis: analyzing historical query data and identifying query patterns; Step S13, partition strategy generation: Generate an optimal partition strategy based on the characteristics and query pattern of a single data source; Step S14: Strategy verification: verify whether the generated partitioning strategy can meet performance and scalability requirements; Step S15, strategy application: applying the generated partition strategy to the data storage layer; Step S16: Automatic tuning: changing the partitioning strategy according to the real-time data characteristics.

[0014] Preferably, in step S2, the partition distribution is automatically adjusted according to the system load to ensure data volume and load balance among the partitions. The specific process is as follows: Step S21: Real-time monitoring and collection of system load data, including: CPU usage, memory usage, disk I / O, network bandwidth, query load, data volume, and log records; Step S22: Evaluate and analyze the collected load data, calculate the load index, and determine the load balancing target; Step S23: Dynamically adjust partition distribution; Step S231: Design a partition migration strategy: First, identify partitions with excessive load and partitions with insufficient load; then, migrate part of the data in the hot partition to the cold partition, that is, redirect query requests to the partition with less load; Step S232, partition splitting and merging: splitting a partition with excessive load into multiple smaller partitions; merging multiple partitions with excessively low load into one partition; Step S233: The above process uses the distributed system automation tool to achieve partition migration. Step S24: Design corresponding load balancing algorithms based on different load balancing modes: (1) Rule-based load balancing: set threshold rules. When the CPU usage of a partition exceeds 80%, data migration is triggered; when the data volume of a partition exceeds 20% of the average value, partition splitting is triggered. (2) Machine learning-based load balancing uses machine learning models to predict load change trends and dynamically adjust partition distribution. It uses historical load data, query patterns, and data distribution as training model inputs to output the optimal partition adjustment strategy. Step S25: Continuously monitor the system load, dynamically adjust the partition distribution, and use the stream processing framework to process the load data in real time; feed the adjusted load data back to the load balancing algorithm to further optimize the partition strategy.

[0015] Preferably, in step S3, a distributed index mechanism and a cache mechanism are introduced to optimize the performance of cross-partition queries. The specific process is as follows: Step S31: Introduce a distributed indexing mechanism to accelerate cross-partition query positioning and data access by establishing indexes on multiple nodes; Step S311, index design: global index, maintain a global index on all nodes, record the location of each data item, that is, partition and node; local index, maintain a local index on each node, record only the data items on the node; hybrid index, combine the advantages of global index and local index, global index is used to quickly locate partitions, local index is used to accelerate queries within partitions; Step S312, index construction: data sharding, sharding the data according to the rules and distributing it to different nodes; index creation, creating the index synchronously when data is written; index update: updating the index synchronously when data is updated or deleted; Step S313, query optimization: query routing, using global indexes to locate the target partition; parallel query: decomposing the cross-partition query into multiple subqueries, executing them in parallel, and finally merging the results; index caching, caching hot indexes in memory, query positioning; Step S32: The goal of introducing a cache mechanism is to cache hot data or query results into memory; Step S321, cache design: query result cache, caching the results of cross-partition queries; data cache, caching hot data to reduce cross-partition data access; distributed cache, using a distributed cache system to store cached data; Step S322, cache strategy: cache update, when data is written, the cache is updated synchronously; when data is written, the cache is invalidated and reloaded the next time it is queried; cache elimination, using the LRU and LFU algorithms to eliminate cold data; cache partitioning, partitioning the cached data according to the query mode; Step S323, query optimization: cache hit, priority access to the cache during query, if the cache hits, the result is returned directly; cache preheating, pre-load hot data into the cache before system startup or peak period; cache sharding, cache data is stored in shards to support parallel query.

[0016] Preferably, in step S4, the number of partitions is dynamically adjusted according to business needs, and data is automatically migrated to adapt to the new partitioning strategy to achieve elastic expansion. The specific process is as follows: Step S41, monitoring and triggering; Step S411: The monitoring system collects data volume, query load, and resource usage of each partition in real time; Step S412: When the data volume or load of a partition exceeds a threshold, the expansion process is triggered; Step S42: partition splitting; Step S421: Split the target partition into multiple sub-partitions according to the splitting rule; Step S422: Distribute the split sub-partitions to different nodes; Step S423: Update metadata to reflect the new partition distribution; Step S43: data migration; Step S431: Migrate the data of the original partition to the new sub-partition; Step S432: During the migration process, lock the partition to ensure data consistency; Step S433: After the migration is completed, release the lock and synchronize the incremental data; Step S44: resource allocation; Step S441: If the existing node resources are insufficient, automatically allocate a new node; Step S442: Distribute the new sub-partition to the new node; Step S45, shrinking process; Step S451: When the data volume or load of multiple partitions is lower than a threshold, a shrink process is triggered; Step S452: Merge the target partitions according to the merging rule; Step S453: Migrate the merged partition data to the target node; Step S454: Release the idle nodes.

[0017] Preferably, in step S5, the system uses an intelligent storage strategy to dynamically adjust the storage of the received single data source based on the access frequency and importance of the single data source. The specific process is as follows: Step S51: The system receives a single data source from a user or an application; Step S52: The single data source is stored in multiple partitions according to the generated partitioning strategy. Each partition independently stores a portion of the data, and multiple copies of the data are stored. Step S53: using data compression technology to reduce storage space usage; Step S54: Automatically update the stored data through the automatic update function to ensure the timeliness of the data; Step S55: Use metadata management to maintain a global metadata table and record information about each partition to facilitate rapid location and query of data.

[0018] A database cutting and storage system based on a single data source includes a data storage layer and a data processing layer; wherein the data storage layer includes multiple partitions, each partition stores a portion of data, communicates with the data processing layer, and is responsible for storage and reading.

[0019] Preferably, the data processing layer includes a partition strategy generation module, a dynamic load balancing module, a query optimization module, an elastic expansion module and a storage management module; (1) Partitioning strategy generation module: Communicates with the data storage layer and generates the optimal partitioning strategy based on the characteristics of the data and query patterns; (2) Dynamic load balancing module: monitors the load of each partition and automatically adjusts the data partition; (3) Query optimization module: optimizes the performance of cross-partition queries; (4) Elastic expansion module: supports dynamic increase or decrease of the number of partitions; (5) Storage management module: responsible for data storage, redundancy, compression and optimization.

[0020] Preferably, during the initialization phase, an initial partitioning strategy (composite partitioning) is generated based on the characteristics of a single data source and business requirements, and the storage engine and storage parameters are configured, and a suitable storage engine is selected. During the operation phase, the system load is monitored in real time, the partition distribution of a single data source is dynamically adjusted, automatic tuning is performed in real time, partition strategies are changed, the performance of cross-partition queries is optimized, and data compression and storage optimization are performed; During the expansion phase, the number of partitions is dynamically increased according to business growth needs, and data is automatically migrated to adapt to the new partitioning strategy to ensure the scalability and load balancing of the system.

[0021] Therefore, the present invention adopts the above-mentioned database segmentation and storage method and system based on a single data source, and the beneficial effects are as follows: (1) Significantly improve query performance: The present invention significantly improves query performance by optimizing cross-partition queries and introducing distributed indexes; (2) Enhanced scalability: The present invention supports dynamic adjustment of the number of partitions according to business needs, significantly enhancing system scalability; (3) Reduce maintenance costs: The present invention reduces the need for manual intervention through automated partitioning strategy generation and dynamic load balancing; (4) Improve storage efficiency: Improve storage utilization and reduce costs through intelligent storage strategies, redundancy optimization, and tiered storage; (5) Enhanced data reliability: Through partitioned storage and multi-copy redundancy, data reliability and fault tolerance are improved.

[0022] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of a database cutting and storage method based on a single data source of the present invention; Figure 2 It is a flow chart of partition strategy generation of the present invention; Figure 3 This is a flow chart of dynamic load balancing and elastic expansion of the present invention; Figure 4 It is a storage flow chart of the present invention; Figure 5 This is an architecture diagram of a database cutting and storage system based on a single data source in the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0025] like Figure 1 As shown, the present invention provides a database cutting and storage method based on a single data source, comprising the following steps: Step S1: Automatically generate the optimal partitioning strategy based on the characteristics and query patterns of a single data source through machine learning algorithms, such as Figure 2 shown.

[0026] Among them, the characteristics of a single data source include: time range, value range, geographical location, etc.; the intelligent partitioning strategy supports multiple methods such as range partitioning, list partitioning, hash partitioning, and composite partitioning (combining time and region partitioning strategies). Figure 2 As shown in the figure, the specific process of partition strategy generation is as follows: Step S11, data characteristic analysis: analyzing the distribution characteristics of a single data source (such as time range, value range, geographical location, etc.).

[0027] Step S12: Historical query pattern analysis: Analyze historical query data to identify common query patterns.

[0028] Step S13, partition strategy generation: Generate the optimal partition strategy (such as range partitioning, list partitioning, hash partitioning, composite partitioning, etc.) based on the characteristics and query mode of the single data source.

[0029] Step S14: Strategy verification: verify whether the generated partitioning strategy can meet performance and scalability requirements.

[0030] Step S15: Strategy application: Apply the generated partition strategy to the data storage layer.

[0031] Step S16: Automatic tuning: changing the partitioning strategy according to the real-time data characteristics.

[0032] Step S2: Automatically adjust the partition distribution according to the system load to ensure data volume and load balance among partitions, and avoid the situation where one partition is overloaded while other partitions are idle.

[0033] Step S21: Real-time monitoring and collection of system load data, specifically including: (1) CPU usage: CPU load of each node.

[0034] (2) Memory usage: the memory usage of each node.

[0035] (3) Disk I / O: disk read and write performance of each node.

[0036] (4) Network bandwidth: network transmission conditions between nodes.

[0037] (5) Query load: query frequency and query response time of each partition.

[0038] (6) Data volume: data size and data distribution of each partition.

[0039] (7) Logging: Recording historical load data for analysis and prediction.

[0040] Step S22: Evaluate and analyze the collected load data.

[0041] (1) Calculate load index: Calculate the load score of each partition, taking into account factors such as CPU, memory, disk I / O, and network bandwidth.

[0042] (2) Determine the load balancing target to ensure that the load score of each partition is close to the average value and that the data volume of each partition is evenly distributed.

[0043] Step S23: Dynamically adjust the partition distribution.

[0044] Step S231, design a partition migration strategy: first identify partitions with excessively high loads (hot spots) and partitions with excessively low loads (cold spots); then, migrate part of the data in the hot spots to the cold spots, i.e. redirect query requests to the partitions with lower loads.

[0045] Step S232, partition splitting and merging: if the load of a partition is too high, split it into multiple smaller partitions; if the load of multiple partitions is too low, merge them into one partition.

[0046] Step S233: The above process uses an automated tool for distributed systems (Apache Kafka) to implement partition migration.

[0047] Step S24: Design corresponding load balancing algorithms based on different load balancing modes.

[0048] (1) Rule-based load balancing: Set threshold rules. If the CPU usage of a partition exceeds 80%, data migration is triggered; if the data volume of a partition exceeds 20% of the average, partition splitting is triggered.

[0049] (2) Machine learning-based load balancing uses machine learning models (such as reinforcement learning) to predict load change trends and dynamically adjust partition distribution; historical load data, query patterns, and data distribution are used as training model inputs to output the optimal partition adjustment strategy.

[0050] Step S25: Continuously monitor the system load, dynamically adjust the partition distribution, and use the stream processing framework (Apache Storm) to process the load data in real time; feed the adjusted load data back to the load balancing algorithm to further optimize the partition strategy.

[0051] Step S3: Introduce a distributed indexing mechanism and a caching mechanism to optimize the performance of cross-partition queries, support parallel queries and distributed caching, and reduce query latency.

[0052] Step S31: The goal of introducing a distributed index mechanism is to accelerate the location and data access of cross-partition queries by establishing indexes on multiple nodes.

[0053] Step S311: Index design.

[0054] Global index: A global index is maintained on all nodes to record the location (partition and node) of each data item. It is suitable for scenarios with high query frequency and wide data distribution.

[0055] Local index: A local index is maintained on each node, recording only the data items on this node. It is suitable for scenarios with a small query range and concentrated data distribution.

[0056] Hybrid index: combines the advantages of global index and local index. Global index is used to quickly locate partitions, and local index is used to accelerate queries within partitions.

[0057] Step S312: Index construction.

[0058] Data sharding: Shard data according to certain rules (such as hash, range) and distribute it to different nodes.

[0059] Index creation: Create indexes synchronously when data is written; use a distributed database or custom indexing service.

[0060] Index update: When data is updated or deleted, the index is updated synchronously.

[0061] Step S313: query optimization.

[0062] Query routing: Use global indexes to quickly locate the target partition and reduce unnecessary cross-partition queries.

[0063] Parallel query: Decompose cross-partition queries into multiple subqueries, execute them in parallel, and finally merge the results.

[0064] Index cache: caches hot indexes in memory to speed up query positioning.

[0065] Step S32: The goal of introducing a cache mechanism is to reduce the latency and system load of cross-partition queries by caching hot data or query results in memory.

[0066] Step S321: cache design.

[0067] Query result cache: caches the results of cross-partition queries to avoid repeated queries. It is suitable for scenarios where query results change infrequently.

[0068] Data caching: Caches hot data to reduce cross-partition data access. It is suitable for scenarios with high data access frequency and small data volume.

[0069] Distributed cache: Use a distributed cache system to store cache data and support high concurrent access.

[0070] Step S322: Cache strategy.

[0071] Cache update: When data is written, the cache is updated synchronously; when data is written, the cache is invalidated and reloaded the next time it is queried.

[0072] Cache elimination: Use LRU, LFU and other algorithms to eliminate cold data.

[0073] Cache partitioning: Partition cache data based on query patterns to prevent a single cache node from becoming a bottleneck.

[0074] Step S323: query optimization.

[0075] Cache hit: When querying, the cache is accessed first, and if the cache hits, the result is returned directly.

[0076] Cache preheating: Preload hot data into the cache before system startup or peak period.

[0077] Cache sharding: Cache data is stored in shards to support parallel query.

[0078] Step S4: Dynamically adjust the number of partitions according to business needs, automatically migrate data to adapt to the new partitioning strategy, achieve elastic expansion, and ensure that the database can flexibly respond to changes in data volume while maintaining efficient data processing capabilities. Figure 3 The specific process of dynamic load balancing and elastic expansion is shown.

[0079] Step S41: monitoring and triggering.

[0080] Step S411: The monitoring system collects data volume, query load and resource usage of each partition in real time.

[0081] Step S412: When the data volume or load of a partition exceeds a threshold, the expansion process is triggered.

[0082] Step S42: partition splitting.

[0083] Step S421: Split the target partition into multiple sub-partitions according to a splitting rule (such as by range or hash).

[0084] Step S422: Distribute the split sub-partitions to different nodes.

[0085] Step S423: Update metadata (such as global index) to reflect the new partition distribution.

[0086] Step S43: data migration.

[0087] Step S431: Migrate the data of the original partition to the new sub-partition.

[0088] Step S432: During the migration process, lock the partition to ensure data consistency.

[0089] Step S433: After the migration is completed, release the lock and synchronize the incremental data.

[0090] Step S44: resource allocation.

[0091] Step S441: If the existing node resources are insufficient, a new node is automatically allocated.

[0092] Step S442: Distribute the new sub-partition to the new node.

[0093] Step S45: shrink process.

[0094] Step S451: When the data volume or load of multiple partitions is lower than a threshold, the shrink process is triggered.

[0095] Step S452: Merge the target partitions according to a merging rule (such as by range or load).

[0096] Step S453: Migrate the merged partition data to the target node.

[0097] Step S454: Release the idle nodes.

[0098] Step S5: The system uses intelligent storage strategies to dynamically adjust the storage of the received single data source based on the access frequency and importance of the single data source. Figure 4 shown.

[0099] Step S51: The system receives a single data source from a user or an application.

[0100] Step S52: The single data source is stored in multiple partitions according to the generated partitioning strategy. Each partition independently stores a portion of data, and multiple copies of the data are stored to improve data reliability and fault tolerance.

[0101] Step S53: Using data compression technology, the storage space occupied is further reduced, thereby improving storage efficiency.

[0102] Step S54: Automatically update the stored data through the automatic update function to ensure the timeliness of the data.

[0103] Step S55: Use metadata management to maintain a global metadata table, record the location, size, data range and other information of each partition, and facilitate rapid location and query of data.

[0104] The system uses intelligent storage strategies to dynamically adjust storage strategies based on the access frequency and importance of a single data source. For example, frequently accessed hot data is stored on high-performance storage media, while infrequently accessed cold data is stored on low-cost storage media, thereby improving storage utilization.

[0105] like Figure 5As shown, the present invention also provides a database segmentation and storage system based on a single data source, which is used to implement the above-mentioned database segmentation and storage method based on a single data source. The system includes a data storage layer and a data processing layer. The data storage layer includes multiple partitions, each of which stores a portion of data and communicates with the data processing layer to be responsible for storage and reading.

[0106] The data processing layer includes a partitioning strategy generation module, a dynamic load balancing module, a query optimization module, an elastic expansion module, and a storage management module.

[0107] (1) Partitioning strategy generation module: Communicates with the data storage layer and generates the optimal partitioning strategy based on the characteristics of the data and the query pattern.

[0108] (2) Dynamic load balancing module: monitors the load of each partition and automatically adjusts the data partition.

[0109] (3) Query optimization module: optimizes the performance of cross-partition queries.

[0110] (4) Elastic expansion module: supports dynamic increase or decrease of the number of partitions.

[0111] (5) Storage management module: responsible for data storage, redundancy, compression and optimization.

[0112] During the initialization phase, the system generates an initial partitioning strategy—composite partitioning (a combination of time range and region list partitioning)—based on the characteristics of a single data source and business needs. It then configures the storage engine and storage parameters, selecting an appropriate storage engine. During the operational phase, the system monitors system load in real time, dynamically adjusts the partition distribution of the single data source, and automatically optimizes the partitioning strategy in real time, changing the partitioning strategy to optimize cross-partition query performance. Data compression and storage optimization are also performed to improve storage efficiency. During the expansion phase, the number of partitions is dynamically increased based on business growth needs, and data is automatically migrated to accommodate the new partitioning strategy. This ensures system scalability and load balancing, significantly improving query performance, enhancing system scalability, and significantly increasing management efficiency while reducing maintenance costs.

[0113] Example Due to information asymmetry in the labor force in remote areas, the mini-program platform needs to store and query massive amounts of order data. Traditional single-table storage methods no longer meet performance requirements. The method proposed in this paper effectively stores data in the database and partitions order data by time range (e.g., month) and region. The number of partitions can be dynamically increased under high load to ensure system performance and scalability. Data storage optimization and automated deletion of expired data ensure data timeliness and readability.

[0114] Therefore, the present invention adopts the above-mentioned database cutting and storage method and system based on a single data source, which significantly improves query performance by optimizing cross-partition queries and introducing distributed indexes; supports dynamic adjustment of the number of partitions according to business needs, significantly enhancing system scalability; through automated partition strategy generation and dynamic load balancing, it reduces the need for manual intervention, thereby reducing maintenance costs, and is suitable for large-scale data storage and query scenarios.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A database cutting and storage method based on a single data source, characterized in that: The following steps are involved: Step S1: Automatically generate the optimal partitioning strategy based on the characteristics and query pattern of a single data source through a machine learning algorithm; Step S2: Automatically adjust the partition distribution according to the system load to ensure data volume and load balance among the partitions; Step S3: Introduce a distributed indexing mechanism and a caching mechanism to optimize the performance of cross-partition queries; Step S4: Dynamically adjust the number of partitions according to business needs, automatically migrate data to adapt to the new partitioning strategy, and achieve elastic expansion; Step S5: The system uses an intelligent storage strategy to dynamically adjust the storage of the received single data source based on the access frequency and importance of the single data source.

2. The database cutting and storage method based on a single data source according to claim 1, characterized in that: In step S1, the characteristics of the single data source include: time range, value range, and geographic location; The intelligent partitioning strategy supports range partitioning, list partitioning, hash partitioning, and composite partitioning.

3. The database cutting and storage method based on a single data source according to claim 1, characterized in that: In step S1, the specific process of partition strategy generation is as follows: Step S11, data characteristic analysis: analyzing the distribution characteristics of a single data source; Step S12, historical query pattern analysis: analyzing historical query data and identifying query patterns; Step S13, partition strategy generation: Generate an optimal partition strategy based on the characteristics and query pattern of a single data source; Step S14: Strategy verification: verify whether the generated partitioning strategy can meet performance and scalability requirements; Step S15, strategy application: applying the generated partition strategy to the data storage layer; Step S16: Automatic tuning: changing the partitioning strategy according to the real-time data characteristics.

4. The database cutting and storage method based on a single data source according to claim 1, characterized in that: In step S2, the partition distribution is automatically adjusted according to the system load to ensure data volume and load balance among the partitions. The specific process is as follows: Step S21: Real-time monitoring and collection of system load data, including: CPU usage, memory usage, disk I / O, network bandwidth, query load, data volume, and log records; Step S22: Evaluate and analyze the collected load data, calculate the load index, and determine the load balancing target; Step S23: Dynamically adjust partition distribution; Step S231: Design a partition migration strategy: First, identify partitions with excessive load and partitions with insufficient load; then, migrate part of the data in the hot partition to the cold partition, that is, redirect query requests to the partition with less load; Step S232, partition splitting and merging: splitting a partition with excessive load into multiple smaller partitions; merging multiple partitions with excessively low load into one partition; Step S233: The above process uses the distributed system automation tool to achieve partition migration. Step S24: Design corresponding load balancing algorithms based on different load balancing modes: (1) Rule-based load balancing: set threshold rules. When the CPU usage of a partition exceeds 80%, data migration is triggered; when the data volume of a partition exceeds 20% of the average value, partition splitting is triggered. (2) Machine learning-based load balancing uses machine learning models to predict load change trends and dynamically adjust partition distribution. It uses historical load data, query patterns, and data distribution as training model inputs to output the optimal partition adjustment strategy. Step S25: Continuously monitor the system load, dynamically adjust the partition distribution, and use the stream processing framework to process the load data in real time; feed the adjusted load data back to the load balancing algorithm to further optimize the partition strategy.

5. The database cutting and storage method based on a single data source according to claim 1, characterized in that: In step S3, a distributed indexing mechanism and a cache mechanism are introduced to optimize the performance of cross-partition queries. The specific process is as follows: Step S31: Introduce a distributed indexing mechanism to accelerate cross-partition query positioning and data access by establishing indexes on multiple nodes; Step S311, index design: global index, maintain a global index on all nodes, record the location of each data item, that is, partition and node; local index, maintain a local index on each node, record only the data items on the node; hybrid index, combine the advantages of global index and local index, global index is used to quickly locate partitions, local index is used to accelerate queries within partitions; Step S312, index construction: data sharding, sharding the data according to the rules and distributing it to different nodes; index creation, creating the index synchronously when data is written; index update: updating the index synchronously when data is updated or deleted; Step S313, query optimization: query routing, using global indexes to locate the target partition; parallel query: decomposing the cross-partition query into multiple subqueries, executing them in parallel, and finally merging the results; index caching, caching hot indexes in memory, query positioning; Step S32: The goal of introducing a cache mechanism is to cache hot data or query results into memory; Step S321, cache design: query result cache, cache cross-partition query results; Data caching, caching hot data to reduce cross-partition data access; distributed caching, using a distributed caching system to store cached data; Step S322, cache strategy: cache update, when data is written, the cache is updated synchronously; when data is written, the cache is invalidated and reloaded the next time it is queried; cache elimination, using the LRU and LFU algorithms to eliminate cold data; cache partitioning, partitioning the cached data according to the query mode; Step S323, query optimization: cache hit, first access the cache during query, and directly return the result if the cache hits; Cache preheating: pre-load hot data into the cache before system startup or peak period; cache sharding: store cache data in shards to support parallel query.

6. The database cutting and storage method based on a single data source according to claim 1, characterized in that: In step S4, the number of partitions is dynamically adjusted according to business needs, and data is automatically migrated to adapt to the new partitioning strategy to achieve elastic expansion. The specific process is as follows: Step S41, monitoring and triggering; Step S411: The monitoring system collects data volume, query load, and resource usage of each partition in real time; Step S412: When the data volume or load of a partition exceeds a threshold, the expansion process is triggered; Step S42: partition splitting; Step S421: Split the target partition into multiple sub-partitions according to the splitting rule; Step S422: Distribute the split sub-partitions to different nodes; Step S423: Update metadata to reflect the new partition distribution; Step S43: data migration; Step S431: Migrate the data of the original partition to the new sub-partition; Step S432: During the migration process, lock the partition to ensure data consistency; Step S433: After the migration is completed, release the lock and synchronize the incremental data; Step S44: resource allocation; Step S441: If the existing node resources are insufficient, automatically allocate a new node; Step S442: Distribute the new sub-partition to the new node; Step S45, shrinking process; Step S451: When the data volume or load of multiple partitions is lower than a threshold, a shrink process is triggered; Step S452: Merge the target partitions according to the merging rule; Step S453: Migrate the merged partition data to the target node; Step S454: Release the idle nodes.

7. The database cutting and storage method based on a single data source according to claim 1, characterized in that: In step S5, the system uses an intelligent storage strategy to dynamically adjust the storage of the received single data source based on the access frequency and importance of the single data source. The specific process is as follows: Step S51: The system receives a single data source from a user or an application; Step S52: The single data source is stored in multiple partitions according to the generated partitioning strategy. Each partition independently stores a portion of the data, and multiple copies of the data are stored. Step S53: using data compression technology to reduce storage space usage; Step S54: Automatically update the stored data through the automatic update function to ensure the timeliness of the data; Step S55: Use metadata management to maintain a global metadata table and record information about each partition to facilitate rapid location and query of data.

8. A database segmentation and storage method based on a single data source according to any one of claims 1 to 6, applied to a database segmentation and storage system based on a single data source, characterized in that: The system includes a data storage layer and a data processing layer; wherein the data storage layer includes multiple partitions, each partition stores a portion of data, communicates with the data processing layer, and is responsible for storage and reading.

9. The database cutting and storage system based on a single data source according to claim 7, characterized in that: The data processing layer includes a partitioning strategy generation module, a dynamic load balancing module, a query optimization module, an elastic expansion module, and a storage management module; (1) Partitioning strategy generation module: Communicates with the data storage layer and generates the optimal partitioning strategy based on the characteristics of the data and query patterns; (2) Dynamic load balancing module: monitors the load of each partition and automatically adjusts the data partition; (3) Query optimization module: optimizes the performance of cross-partition queries; (4) Elastic expansion module: supports dynamic increase or decrease of the number of partitions; (5) Storage management module: responsible for data storage, redundancy, compression and optimization.

10. The database cutting and storage system based on a single data source according to claim 7, characterized in that: During the initialization phase, based on the characteristics of a single data source and business requirements, an initial partitioning strategy (composite partitioning) is generated, the storage engine and storage parameters are configured, and an appropriate storage engine is selected. During the operation phase, the system load is monitored in real time, the partition distribution of a single data source is dynamically adjusted, automatic tuning is performed in real time, partition strategies are changed, the performance of cross-partition queries is optimized, and data compression and storage optimization are performed; During the expansion phase, the number of partitions is dynamically increased according to business growth needs, and data is automatically migrated to adapt to the new partitioning strategy to ensure the scalability and load balancing of the system.

Images