Dynamic data sub-library and sub-table strategy generation method and system based on large model driving

Abstract

The invention relates to the technical field of big data. The dynamic data sub-library and sub-table strategy generation method based on large model driving comprises the steps of collecting operation indexes of a database and business, and obtaining a business data access mode portrait in combination with business topology and consistency requirements; according to the business data access mode portrait, obtaining a candidate sub-library sub-table strategy scheme through a large language model reasoning technology; performing risk assessment on the candidate sub-library and sub-table strategy schemes to generate a strategy verification list and a risk prompt; according to the strategy verification list and the risk prompt, an executable operation and maintenance operation manual is obtained through a migration path planning technology, and the executable operation and maintenance operation manual is used for being manually executed by a database administrator DBA; obtaining strategy implementation effect monitoring data according to the implementation effect of the executable operation and maintenance operation manual; and performing database and table division rule constraint extraction on the strategy implementation effect monitoring data to generate a strategy knowledge base.

Description

A method and system for generating dynamic database sharding and table partitioning strategies based on large models Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a method and system for generating dynamic data sharding and partitioning strategies based on a large model. Background Technology

[0002] As enterprises deepen their digital transformation, the volume of business data is surging, and traditional single-machine databases are facing bottlenecks in terms of capacity and concurrency, making database sharding and table partitioning an inevitable choice. However, traditional database sharding and table partitioning strategies are usually formulated once before the system goes live based on static rules (such as by time or ID hash), which is difficult to adapt to dynamic changes in business.

[0003] In actual operation, as business models evolve, user behavior changes, and new channels are integrated, the original static sharding strategy gradually becomes ineffective, leading to performance issues such as data skew, access hotspots, and a surge in cross-database queries. At this point, relying on manual intervention by database administrators (DBAs) to adjust the strategy is not only slow and costly, but also suffers from a series of problems, including subjective judgment, uncontrollable implementation risks, and a lack of standardized procedures. Crucially, manual adjustments lack a deep understanding of business semantics (such as microservice call chains and transaction consistency requirements), making it difficult to generate sharding solutions that accurately match real-world business scenarios, and also preventing the effective accumulation and reuse of experience from each adjustment for subsequent optimizations.

[0004] Therefore, enterprises urgently need a dynamic database sharding and table partitioning solution generation method that can sense business status, generate targeted sharding strategies, and continuously optimize itself, in order to solve the core contradiction of the lag of static strategies and the inefficiency of manual optimization. Summary of the Invention

[0005] The main purpose of this application is to provide a method for generating dynamic data sharding and table partitioning strategies based on a large model, so as to solve the technical problems mentioned in the background.

[0006] To achieve the above objectives, the first aspect of this application proposes a method for generating dynamic data sharding and partitioning strategies based on a large model, comprising: collecting operational metrics of the database and business operations, and combining these metrics with business topology and consistency requirements to obtain a business data access pattern profile; obtaining candidate sharding and partitioning strategy schemes based on the business data access pattern profile using large language model inference technology; conducting risk assessments on the candidate sharding and partitioning strategy schemes to generate a strategy verification list and risk warnings; obtaining an executable operation and maintenance manual based on the strategy verification list and risk warnings using migration path planning technology, wherein the migration path planning technology includes migration steps, gray-scale switching strategies, transaction assurance guidelines, and rollback contingency plans, and the executable operation and maintenance manual is for manual execution by database administrators (DBAs); obtaining strategy implementation effect monitoring data based on the implementation effect of the executable operation and maintenance manual; and extracting sharding and partitioning rule constraints from the strategy implementation effect monitoring data to generate a strategy knowledge base, wherein the sharding and partitioning rule constraints in the strategy knowledge base are used as one of the input conditions for generating the business data access pattern profile in subsequent steps of obtaining the business data access pattern profile.

[0007] In some feasible methods, the step of collecting database and business operation metrics, combined with business topology and consistency requirements, to obtain a business data access pattern profile includes: collecting table structure metadata of the database; collecting database-side operation metrics to obtain a first metric set, wherein the first metric set includes table capacity, single table growth rate, frequency of inserts, deletes, and modifications, index hit rate, lock wait status, and cross-database transaction distribution; collecting application-side operation metrics of the database to obtain a second metric set, wherein the second metric set includes interface response time, P99 latency, typical query statements, and transaction call chain span; and analyzing the data based on the first and second metric sets. The call relationships between microservices and the data tables that each service depends on are used to generate business topology relationships, as well as consistency requirements and constraints on these relationships. The table structure metadata, the first set of indicators, the second set of indicators, the business topology relationships, and the consistency requirements and constraints on these relationships are then integrated to generate a structured business data access pattern profile. This profile includes at least table structure information obtained from the table structure metadata, access pattern characteristics and data growth trends derived from analyzing the first and second sets of indicators, and business priority information derived from analyzing the business topology relationships and consistency requirements and constraints.

[0008] In some feasible methods, the step of obtaining candidate database sharding and table partitioning strategies based on the business data access pattern profile and through large language model reasoning technology includes: encapsulating the structured information in the business data access pattern profile into standardized prompts; inputting the standardized prompts into a large language model, and generating candidate database sharding and table partitioning strategies through semantic analysis, wherein the candidate database sharding and table partitioning strategies include recommended sharding types, reasons for selecting sharding keys, suggestions for hot data isolation, and consistent access path planning.

[0009] In some feasible approaches, the step of performing a risk assessment on the candidate database sharding and table partitioning strategy and generating a strategy verification list and risk warnings includes: generating a strategy verification list based on the candidate database sharding and table partitioning strategy, wherein the strategy verification list includes typical query and transaction scenarios that need to be replayed, key performance indicators that need to be observed, and their thresholds; and generating risk warnings based on the candidate database sharding and table partitioning strategy, wherein the risk warnings include predictions of potential performance bottlenecks and consistency risks.

[0010] In some feasible methods, the step of obtaining an executable operation and maintenance manual based on the strategy verification checklist and risk warnings through migration path planning technology includes: generating a data migration step plan based on the candidate database sharding and table partitioning strategy scheme, the strategy verification checklist, and the risk warnings. The data migration step plan includes: recommending priority business lines or channels for migration, the time window or data range for batch migration, and the method for handling newly written data during migration; generating a routing strategy and a canary switchover strategy based on the candidate database sharding and table partitioning strategy scheme and the risk warnings. The routing strategy includes rules for determining data routing paths based on sharding keys, and the canary switchover strategy includes the order of gradually switching traffic proportions, the interfaces and metrics that need to be monitored after the switchover, and a full switchover. The following steps are required: First, determine the conditions that must be met. Second, based on the business topology and consistency requirements in the business data access pattern profile, generate consistency and transaction assurance guidelines. These guidelines include requirements for maintaining the same database query path for consistency links and recommendations for retaining the old database as the primary data source during the migration transition period for accounting scenarios. Third, based on the risk warnings, generate a rollback plan. This plan includes rollback trigger conditions, rollback step sequence, and data consistency checkpoints after rollback. Fourth, integrate the data migration step plan, routing and canary switching strategies, consistency and transaction assurance guidelines, and rollback plan to obtain an executable operation manual. This manual is for database administrators (DBAs) to manually implement database sharding and table partitioning schemes.

[0011] In some feasible methods, the step of obtaining strategy implementation effect monitoring data based on the implementation effect of the executable operation and maintenance manual includes: after the database administrator (DBA) completes the execution of the database sharding and table partitioning scheme according to the executable operation and maintenance manual, the DBA collects key performance indicator data during runtime to obtain strategy implementation effect monitoring data, wherein the key performance indicator data are runtime indicators used to evaluate the implementation effect of the candidate database sharding and table partitioning strategy scheme.

[0012] In some feasible methods, the step of extracting sharding and table partitioning rule constraints from the strategy implementation effect monitoring data to generate a strategy knowledge base includes: comparing and analyzing the strategy implementation effect monitoring data with the expected effects of the candidate sharding and table partitioning strategy schemes to generate a strategy implementation effect report; extracting sharding and table partitioning rules and constraints from the strategy implementation effect report, and storing the extracted sharding and table partitioning rules and constraints in the strategy knowledge base. The strategy knowledge base is used as one of the input conditions in the subsequent step of obtaining a business data access pattern profile by combining the operational indicators of the collected database and business with business topology and consistency requirements.

[0013] Secondly, this application provides a dynamic data sharding and partitioning strategy generation system based on a large model, applied to the aforementioned dynamic data sharding and partitioning strategy generation method based on a large model. The system includes: a collection unit for collecting operational metrics of the database and business processes, and combining these metrics with business topology and consistency requirements to obtain a business data access pattern profile; an inference unit for obtaining candidate sharding and partitioning strategy schemes based on the business data access pattern profile using large language model inference technology; a generation unit for performing risk assessment on the candidate sharding and partitioning strategy schemes, and generating a strategy verification list and risk warnings; and a processing unit for processing the strategy verification list and risk warnings using migration path planning technology. The system employs a technique to obtain an executable operation and maintenance manual. The migration path planning technique includes migration steps, canary deployment strategies, transaction assurance guidelines, and rollback contingency plans. This executable operation and maintenance manual is for manual execution by database administrators (DBAs). An effect monitoring unit is used to obtain strategy implementation effect monitoring data based on the implementation effect of the executable operation and maintenance manual. A result unit is used to extract sharding and partitioning rule constraints from the strategy implementation effect monitoring data to generate a strategy knowledge base. In the subsequent step of obtaining a business data access pattern profile, the sharding and partitioning rule constraints in the strategy knowledge base are used as one of the input conditions for generating the business data access pattern profile.

[0014] Thirdly, this application provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.

[0015] Fourthly, this application provides a computer program that, when executed by a processor, implements the steps of the aforementioned method.

[0016] The technical solutions provided by the embodiments of this application can include the following beneficial effects: In the dynamic data sharding and table partitioning strategy generation method based on a large model driven by this application, by introducing a large language model to jointly analyze business semantics and operational indicators, a sharding strategy that is deeply compatible with specific business scenarios can be generated, rather than a templated solution. At the same time, the strategy generation, risk assessment, and migration planning are integrated into a directly implementable manual, which significantly reduces the decision-making and implementation threshold for DBAs. More importantly, the implementation effect is precipitated into a reusable strategy knowledge base through a closed-loop feedback mechanism, enabling the system to have the ability to continuously learn and adaptively optimize. This effectively solves the core pain point that traditional static sharding strategies are difficult to adapt to dynamic business changes, and realizes a paradigm shift from one-time static planning to continuous dynamic optimization. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of the accompanying drawings are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 is a flowchart of a method for generating dynamic data sharding and table partitioning strategies based on a large model driven by this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0021] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0022] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0023] The following explains some terms used in this application to help understand the technical solution. Terms not listed are interpreted using industry terminology and are not considered limiting in this application: APM stands for Application Performance Management or Application Performance Monitoring, referring to the activities and tools used to monitor and manage the performance and availability of software applications. Its goal is to ensure that applications meet user experience requirements and to quickly locate and resolve performance issues.

[0024] A microservice tracing system (also known as a distributed tracing system) is a tool used to collect, store, analyze, and visualize the complete call chain of each user request in a distributed software architecture composed of microservices. Its core value lies in its ability to clearly display the complete request path, the time spent within each service, and the call relationships between services when a request (such as a user placing an order) needs to flow through multiple microservices (such as order service, inventory service, and payment service). This allows for rapid identification of performance bottlenecks and root causes of failures. SkyWalking and Jaeger are both open-source application performance monitoring (APM) and distributed tracing systems. Their main function is to collect, store, analyze, and visualize the complete call chain path of each user request in a distributed system composed of microservice architectures.

[0025] A gateway, or load balancer, is a core network infrastructure whose function is to distribute incoming network connections or data requests to multiple identical backend service instances according to a predetermined strategy. Its main goals are to achieve traffic distribution, eliminate single points of failure, and improve the overall throughput and availability of the system.

[0026] Gateway (specifically API gateway): It serves as the unified traffic entry point in modern application architectures, with all external requests first reaching the gateway. Beyond basic routing functions, its core value lies in providing cross-domain traffic governance, security, and control capabilities, such as authentication, authorization, rate limiting, monitoring, and protocol conversion.

[0027] A distributed tracing system is a set of tools used to collect, store, analyze, and visualize the complete call chain of every user request in a distributed software architecture composed of microservices, databases, caches, and other components.

[0028] Microservices are an architectural style that builds a single large application (monolithic application) into a set of small, independent, loosely coupled services organized around business capabilities. Each service runs in its own process and collaborates through lightweight communication mechanisms such as HTTP / REST or RPC. Each service focuses on completing a specific business task and can be developed, deployed, scaled, and iterated independently.

[0029] ACID, Atomicity, Consistency, Isolation, and Durability, are a set of criteria defining four key properties that determine whether database transactions can be reliably processed. A database system that supports ACID ensures that a series of database operations (a transaction) remains safe and predictable under concurrent operations and potential failures. Atomicity: All operations within a transaction either complete entirely or not at all; it cannot end prematurely. If an error occurs during transaction execution, it will be rolled back to the state before the transaction began. Consistency: The result of a transaction must transform the database from one consistent state to another. That is, transactions must adhere to all defined rules in the database (such as constraints, triggers, cascading, etc.) to ensure data integrity and logical correctness. Isolation: The database allows multiple transactions to execute concurrently, but isolation ensures that concurrent transactions do not interfere with each other; each transaction is unaware that other transactions are executing simultaneously. Databases achieve this by setting different transaction isolation levels (such as Read Uncommitted, Read Committed, Repeatable Read, and Serializable). Durability means that once a transaction is successfully committed, the changes it makes to the database are permanent, and the data will not be lost even if the system subsequently fails (such as a power outage or crash).

[0030] As shown in Figure 1, in the first aspect, this application provides a method for generating dynamic data sharding and partitioning strategies based on a large model, including: S100, collecting operational metrics of the database and business, and combining business topology and consistency requirements to obtain a profile of business data access patterns.

[0031] Specifically, obtaining a business data access pattern profile may include the following steps: S101, collecting the table structure metadata of the database.

[0032] Specifically, in step S101, the definition information of the target database table is automatically obtained from the data dictionary or system directory of the database management system such as MySQL or Oracle by executing a specific metadata query command (such as the INFORMATION_SCHEMA query in MySQL).

[0033] For example, the metadata collected in the database should include at least the following: table name, field name of each field, data type (e.g., int, varchar), length constraint, whether nullables are allowed, default value, primary key constraint (including the fields that constitute the primary key), foreign key constraint (including the referenced tables and fields), index information (including index name, index type such as B-Tree / Hash, the fields contained in the index and their order), table engine type (e.g., InnoDB), and table character set.

[0034] Obtaining this metadata is to understand the static structure of the data in subsequent steps. For example, primary key fields are important candidates for subsequent shard key selection, while foreign key relationships imply the associations between tables, which can affect whether related tables should be placed in the same shard to reduce cross-database queries.

[0035] S102, Collect the database-side operating indicators to obtain the first indicator set.

[0036] The first set of indicators includes table capacity, single table growth rate, frequency of inserts, deletes, and modifications, index hit rate, lock wait status, and cross-database transaction distribution.

[0037] Specifically, in step S102, by connecting to the database instance, periodically querying its performance views (such as Oracle's V$ view), status variables (such as MySQL's SHOW GLOBAL STATUS), or reading the database slow query log and error log, quantitative indicators reflecting the internal operating status of the database are collected.

[0038] For example: Table capacity and growth rate: By periodically (e.g., every minute) querying the amount of data (number of rows) and the storage space occupied (e.g., MB) of the table, the growth rate per unit time is calculated to predict future storage needs and identify large tables that may become bottlenecks due to rapid growth.

[0039] CRUD frequency: Assess table write pressure by monitoring database transaction logs or counting the number of INSERT, UPDATE, and DELETE statements. Tables with high write frequency require higher write scalability from sharding strategies.

[0040] Index hit rate: Calculated as (number of index reads) / (number of index reads + number of full table scans). This index hit rate metric is used to determine the effectiveness of existing indexes. A low hit rate indicates that queries are not effectively utilizing the index, which may lead to performance issues. When planning sharding, it is necessary to consider optimizing the index or choosing a more suitable sharding key.

[0041] Lock wait status: Monitor the number of row lock waits and table lock waits, as well as the average wait time. High lock waits indicate concurrent write conflicts and are a key indicator for identifying write hotspots. This needs to be mitigated by distributing the write load across different database instances through sharding.

[0042] Cross-database transaction distribution: In a sharded database, transaction information recorded by the database broker or application layer is used to statistically analyze the proportion and frequency of distributed transactions involving multiple shards (databases). A high proportion of cross-database transactions can severely impact performance; one of the goals of sharding strategies is to minimize such transactions.

[0043] S103, collect the application-side operation indicators of the database to obtain the second indicator set.

[0044] The second set of metrics includes interface response time, P99 latency, typical query statements, and transaction call chain span.

[0045] Specifically, step S103 collects database access behavior observed from the application's perspective by integrating application performance management (APM) tools, microservice tracing systems (such as SkyWalking, Jaeger), and access logs from gateways or load balancers.

[0046] For example: Interface response time and P99 latency: Obtain response time statistics for each business interface that calls the database from APM, especially the P99 latency (i.e., 99% of requests are completed within this time), to identify the slow query paths with the worst user experience and the greatest need for optimization.

[0047] Typical query statements: Capture the actual SQL statements sent to the database by the application using interceptors in the database driver layer or APM tools (these statements need to be anonymized). Analyze the patterns in these statements (such as commonly used fields in WHERE conditions, JOIN types and tables, and fields in GROUP BY and ORDER BY) to provide direct evidence for selecting sharding keys and determining whether cross-shard queries will occur.

[0048] Transaction call chain span: This refers to the trace data obtained from the distributed tracing system, showing the transaction chain triggered by a complete business request, spanning multiple microservices, and ultimately involving one or more database operations. Analyzing this chain clarifies: Service-data dependencies: which services operate on which database tables; Transaction boundaries: which database operations must be completed within the same database transaction (strong consistency requirement) within a single business operation, which determines that the data involved in these operations should ideally reside on the same shard.

[0049] S104. Based on the first indicator set and the second indicator set, parse the call relationship between microservices and the data tables that each service depends on, generate the business topology relationship, and the consistency requirements constraint on the business topology relationship.

[0050] Specifically, step S104 is an analytical reasoning process that integrates multi-source information. It comprehensively utilizes operational metrics from both the database side (first set of metrics) and the application side (second set of metrics) to cross-validate, quantify, and deepen the understanding of business topology and consistency requirements. This transforms raw, isolated data points into topological relationships and constraint rules with business semantics.

[0051] For example, the initial topology discovery is performed by using the transaction call chain span and typical query statements in the second set of indicators as the main clues. By parsing this data, the system can automatically depict the call paths between microservices and map specific database table access operations to the services that initiate them, thus forming a preliminary "service-data dependency graph". For example, the system can discover that when the "order service" creates an order, it will call the "inventory service" and the "user service" in sequence, and finally perform a write operation on the orders table.

[0052] However, the initial topological relationships only reveal static associations. To imbue this graph with dynamic and quantifiable business meaning, a first set of metrics needs to be introduced for in-depth verification and enhancement. For example, when the second set of metrics shows that the "Order Service" frequently reads and writes to the `orders` table, the extremely high frequency of inserts, deletes, and updates to this table and the significant lock waits in the first set of metrics provide empirical evidence from the database side for this access behavior. This not only confirms that the `orders` table is a core dependency of the Order Service but also quantifies its access pressure, indicating that this dependency is high-frequency and performance-sensitive. Furthermore, the cross-database transaction distribution metrics in the first set of metrics play a crucial role. If the `orders` table and the `order_details` table are frequently observed in the same cross-database transaction, even with complex call chain logic, this constitutes strong evidence that these two tables have a close transactional relationship in terms of business, which must be given special consideration during sharding.

[0053] Based on the above, the final structured output of business topology relationships and consistency requirements is no longer a simple logical deduction, but a decision-making basis supported by runtime data. The topology relationships will mark core, high-frequency dependency paths; consistency constraints will explicitly indicate strong consistency requirements (such as orders and order_details must be deployed in the same shard) and their priorities (quantified through associated latency and transaction costs). This process essentially verifies the internal operating state of the database with the business logic of the application layer, thereby generating a "business data relationship graph" that far exceeds the static architecture diagram and truly reflects the runtime characteristics of the system, laying the foundation for generating accurate sharding strategies for subsequent large-scale models.

[0054] S105, integrate the table structure metadata, the first indicator set, the second indicator set, the business topology relationship, and the consistency requirements and constraints on the business topology relationship to generate a structured business data access pattern profile.

[0055] The business data access pattern profile includes at least the table structure information obtained from the table structure metadata, the access pattern characteristics and data growth trends derived by analyzing the first indicator set and the second indicator set, and the business priority information derived by analyzing the business topology relationship and consistency requirement constraints.

[0056] Specifically, step S105 is an information fusion and structured description process. Its purpose is to integrate all the heterogeneous information collected in the first four steps into a unified data structure that can comprehensively describe the current system state, providing high-quality input for subsequent Large Language Model (LLM) analysis.

[0057] For example, a standardized data structure (e.g., a JSON or XML template) can be created, information from different sources can be populated into corresponding fields, and relationships between them can be established.

[0058] For example, for the orders table: fill in its table structure information (fields, primary key, etc.) from S101.

[0059] Fill in the data growth trend (current capacity, average daily growth) and access pattern characteristics (such as high write frequency, existence of lock wait) from S102.

[0060] Fill in its business priority information from S103 and S104: it is dependent on "Order Service" and "Payment Service" (business topology), and has strong consistency requirements (consistency constraints) with the accounting_flows table in payment transactions.

[0061] Generate profile content: Table structure information: Metadata directly from S101.

[0062] Access pattern characteristics and data growth trends: These are conclusive descriptions derived from aggregating and calculating time-series metrics data in S102 and S103 (such as calculating growth rates and identifying QPS peaks). For example, "The user_profile table mainly exhibits a read-heavy, write-light pattern, with peak batch updates occurring every day in the early morning."

[0063] Business priority information: Based on the analysis results of S104, each table or business process is marked. For example, "The orders table and the accounting_flows table have strong consistency requirements in the 'payment' business process, and the priority is 'high'."

[0064] The final "Business Data Access Pattern Profile" is a structured document. It is no longer the original monitoring data points, but a report with business semantics that has undergone preliminary analysis and integration, and directly serves the generation of LLM policies in the S200 stage.

[0065] S200: Based on the business data access pattern profile, candidate database sharding and table partitioning strategies are obtained through large language model inference technology.

[0066] Specifically, obtaining a candidate database sharding and table partitioning strategy may include the following steps: S201, encapsulating the structured information in the business data access pattern profile into standardized prompts.

[0067] Specifically, in step S201, the business data access pattern profile generated in step S100 is converted into an instruction format that the big oracle model can understand and process, namely "standardized prompts".

[0068] For example, information in the business data access pattern profile is categorized and organized according to a preset template. This template aims to guide the LLM to think like a database administrator (DBA). For example, the prompt template may include the following core parts: background and current situation, describing the current system's sharding method, data scale, growth trend, and identified performance bottlenecks (such as "Currently, sharding is done by user ID hash, but this causes the shards of several large customers to become hotspots").

[0069] Business models and constraints, clarifying key access patterns (such as "order queries need to be associated with user information at the same time"), strong consistency requirements (such as "payment record generation and account balance update must be completed within the same transaction"), and other business rules.

[0070] The optimization objectives clarify the core issues that need to be addressed in this strategy generation, such as "alleviating access hotspots caused by top customers" and "reducing cross-database query latency caused by the asynchrony of order and user information sharding".

[0071] Step S201 involves a format conversion from the internal data structure to the large language model's interactive interface. Its input is a business data access pattern profile generated by upstream steps, containing elements such as table structure, operational metrics, and business topology. This profile is typically in a machine-readable structured data format (such as a JSON object). S201's task is not to create new content, but rather to act as a "translator" or "adapter," reorganizing and filling in all the key information contained in this profile according to a pre-defined text template that the large language model can understand. This constructs a complete "standardized hint" containing the task context, constraints, and optimization goals. The core of this process is ensuring that information is transmitted to the large language model completely, accurately, and unambiguously, laying the foundation for subsequent reasoning.

[0072] S202, the standardized prompts are input into the large language model, and candidate database sharding and table partitioning strategies are generated through semantic analysis.

[0073] The candidate database sharding and table partitioning strategy includes recommended sharding types, reasons for selecting sharding keys, suggestions for hot data isolation, and planning of consistent access paths.

[0074] Specifically, the essence of step S202 is to use the general knowledge base and logical reasoning capabilities of the Big Prophecy model to solve problems in specific professional fields.

[0075] Once standardized prompts are input, the large language model initiates a deep semantic understanding process. It identifies descriptions of the current system state, performance bottlenecks, business rules, and constraints, and performs association, reasoning, and decision-making based on pre-trained knowledge. For example, when the model simultaneously understands the constraints of "access hotspots" and "strong consistency transaction requirements," it comprehensively utilizes its knowledge of sharding strategies and database transaction principles to generate a technical solution that balances multiple factors. The output of this solution, such as sharding type selection, sharding key justification, hotspot isolation methods, and consistency assurance paths, is the result of the large language model's creative reasoning after applying general knowledge to the specific business data profile. Ultimately, the solution generated by the large language model and presented in structured text form is the "candidate database sharding strategy solution" for DBA evaluation.

[0076] It should be noted that the following explanations are provided regarding the candidate database sharding and table partitioning strategies, including recommended sharding types, rationale for sharding key selection, hotspot data isolation suggestions, and consistent access path planning: Recommended sharding types refer to the specific data splitting methodology recommended by the large language model based on its semantic understanding of business data access patterns. This includes the sharding algorithm or strategy used, such as range-based sharding (e.g., partitioning by time interval), hash-based sharding (e.g., using user ID hash modulo), list-based sharding (e.g., partitioning by explicitly enumerated channels or regions), or a combination of multiple sharding types (e.g., first partitioning the database by time range, then partitioning tables within the database by hash). The purpose of recommending sharding types is to most effectively match data access patterns, thereby optimizing query performance and distribution uniformity.

[0077] The rationale for choosing a sharding key should explain the business logic and technical justification for selecting one or more business fields (such as user_id, order_time) as the sharding key during the decision-making process for the large language model. For example, the reason for choosing "tenant ID" as the sharding key might be: "Because the business access pattern shows that the vast majority of queries are performed within a tenant, choosing this key ensures that data within a tenant is naturally clustered in the same shard, minimizing cross-shard join queries and ensuring query performance." This provides the DBA with a transparent chain of reasoning for decision-making.

[0078] Hotspot data isolation is a targeted solution proposed by the large language model after identifying potential or existing hotspots (such as top users or high-traffic channels) to address the issue of access hotspots and data skew. For example: "It is identified that channel A accounts for 70% of the total access volume. It is recommended to independently split its data into dedicated physical shards to achieve resource isolation and prevent its traffic fluctuations from affecting the services of other channels."

[0079] Consistent access path planning refers to the data distribution and access rules designed by the large language model to ensure strong consistency requirements in critical business processes. For example, for a strong consistency transaction such as "updating the order status and recording the transaction history after payment," the model would plan: "The order table and the transaction history table should be sharded using the same sharding key (such as order_id) to ensure that they are always located within the same database shard. This allows the transaction to be completed through a local transaction, avoiding complex distributed transactions and ensuring ACID properties at the sharding level."

[0080] It should be noted that the recommended sharding types, rationale for sharding key selection, hotspot data isolation suggestions, and consistent access path planning included in the candidate database sharding and table partitioning strategies provide an explanation of the strategy. This explanation is derived from the Big Oracle model, thus facilitating manual execution by database administrators (DBAs). The Big Oracle model involved in this application does not involve any improvements to its structural framework.

[0081] S300, perform a risk assessment on the candidate database sharding and table partitioning strategy schemes, and then generate a strategy verification list and risk warnings.

[0082] Specifically, generating a strategy verification list and risk warning may include the following steps: S301, generating a strategy verification list based on the candidate database sharding and table partitioning strategy scheme.

[0083] The strategy verification list includes typical query and transaction scenarios that need to be replayed, key performance indicators that need to be observed, and their thresholds.

[0084] Specifically, the system receives the structured candidate sharding and partitioning strategies output from step S200. These strategies may include fields such as `sharding_type` (sharding type, e.g., "hash_by_user_id"), `sharding_key` (sharding key, e.g., "user_id"), and `hotspot_mitigation` (hotspot mitigation measures, e.g., "isolate_top_users"). Next, key features are extracted to form a feature vector F = [f1, f2, ..., fn], where each f represents a strategy feature, such as `is_hotspot_mitigation_present` (whether hotspot mitigation exists) and `is_cross_shard_join_required` (whether cross-shard joins are likely to occur).

[0085] Establish a predefined validation rule base. Each rule is a condition and action pair: the condition part is based on the policy feature F and makes a logical judgment (e.g., IFsharding_type=="range_by_time" AND sharding_key=="create_time").

[0086] Action section: Specify the validation scenario template and metric template to be added. The templates are parameterized. For example, the scenario template T_scenario may be "Replay the exact query based on [field name]", and the metric template T_metric may be "Observe the P99 latency of [operation name], the threshold should be lower than [baseline value]*1.1".

[0087] Next, the feature vector F is matched against the conditions of all rules R. For all triggered rules, the template specified in the action portion will be selected.

[0088] For each selected template, the system uses specific parameters from the candidate database sharding strategy (such as the specific value of sharding_key) to populate and generate specific validation items. For example, if a rule triggers a scenario template of "testing sharding key query", the system will instantiate it as "replaying an exact query based on user_id".

[0089] It should be noted that the generation of the strategy verification list requires a comprehensive analysis of candidate database sharding and table partitioning strategies and business data access pattern profiles.

[0090] Test scenario source: The system directly extracts key typical query and transaction scenarios (such as high-concurrency order placement, real-time customer service query, and accounting reconciliation) from the business data access pattern profile generated by step S100. These scenarios serve as the benchmark for verifying the impact of any strategy on existing business.

[0091] Evaluation Item Generation: The system parses candidate database sharding and table partitioning strategies to form a feature vector F. Then, F is matched against a predefined validation rule base. Successfully matched rules instruct the system to bind the benchmark test scenario extracted in step 100 with specific performance metrics and thresholds, thereby generating specific validation items. For example, when a rule in the rule base regarding "range sharding" is triggered, it instructs the system to add the observation metric of "cross-shard scan count" to the benchmark scenario of "real-time customer service query".

[0092] Furthermore, if the solution aims to optimize user order queries, then "querying all user orders" will be listed as a verification scenario. Simultaneously, the system will set key performance indicators (KPIs) to be observed for each verification scenario and specify a specific threshold for each KPI. This threshold serves as the objective basis for determining whether the test passes. For example, for the "querying all user orders" scenario, the system will set the "request response time" KPI and clearly define its threshold. These scenarios, KPIs, and thresholds together constitute the strategy verification checklist, providing clear testing objectives and acceptance criteria for subsequent verification in the pre-release environment.

[0093] S302, Based on the candidate database sharding and table partitioning strategy, generate a risk warning.

[0094] The risk warnings include predictions of potential performance bottlenecks and consistency risks.

[0095] Specifically, based on the candidate database sharding and table partitioning strategies, we analyze whether the sharding rules might lead to excessive concentration of data or access traffic on certain shards. For example, if sharding is done by date, the shard with the most recent date might bear the majority of the write pressure, forming a new hotspot.

[0096] Consistency risk prediction involves the system checking whether transactions that could have been easily completed within a single database will become complex and error-prone cross-shard transactions due to the distribution of related data across different shards under the new data distribution.

[0097] The above method yields a series of risk warning items, each explaining the nature of the potential problem (whether it's a performance bottleneck or a consistency risk) and its possible causes. These risk warnings, along with the policy verification checklist, will serve as the core focus and decision-making basis for DBAs when validating candidate solutions in a pre-release environment.

[0098] It should be noted that the process of generating risk warnings involves comparing and matching candidate database sharding and table partitioning strategies with actual business operation records. First, key features of the candidate strategies (such as sharding keys and sharding methods) are extracted. Second, these key features are matched against conditions in a pre-built rule base (rules in the rule base can be derived from historical data summaries or expert advice) (e.g., the rule "If the sharding key ≠ transaction association key, then a consistency risk is triggered"). Finally, for successfully matched rules, their corresponding risk warning entries are output. The entire process is an automated risk deduction achieved through feature matching and rule triggering.

[0099] Based on the policy verification checklist and risk warnings, S400 uses migration path planning technology to generate an executable operation and maintenance manual.

[0100] The migration path planning technology includes migration steps, gray-scale switching strategies, transaction protection guidelines, and rollback contingency plans. The executable operation manual is for database administrators (DBAs) to execute manually.

[0101] Specifically, obtaining an executable operation and maintenance manual may include the following steps: S401, generating a data migration step plan based on the candidate database sharding and table partitioning strategy scheme, the strategy verification list, and the risk warning.

[0102] The data migration plan includes: recommending priority business lines or channels for migration, time windows or data ranges for phased migration, and methods for handling newly written data during the migration process.

[0103] Specifically, the decision-making basis for prioritizing the migration of business lines or channels is the risk level predicted in the risk warning. The system will prioritize the migration of those with relatively low business importance, those expected to have the most significant performance benefits under the new strategy, or those judged to have relatively low risk based on the risk warning. This follows the principle of prioritizing simple, low-risk components. For example, a threshold can be set and used as a benchmark for comparison to determine the magnitude of the risk, thereby identifying the business lines or channels to be migrated first.

[0104] The decision to migrate data in batches, including the time window or data range, is based on the characteristics and cyclical patterns of the business data. For example, migration can be carried out during off-peak hours to reduce disruption; batch migration can be divided according to time sequence (e.g., starting with older data) or specific logical range (e.g., data belonging to a specific supplier or region) to facilitate gradual verification and problem localization. For instance, rules for batch migration can be constructed, such as migrating data with cyclical patterns or data older than one year in chronological order first, and based on these rules, time windows or data ranges for batch migration can be defined.

[0105] To ensure data consistency during migration, the handling of newly written data can be achieved through several methods. One approach is to briefly intercept write requests for newly generated data within a specific data range during migration (this requires the business system to allow for a brief interruption). This ensures the static consistency of the migrated data, preventing inconsistencies or data loss due to ongoing modifications to the source data. By temporarily intercepting new writes to this data range, it ensures the range is in a stable state at the start of the migration, allowing the migration tool to capture and copy a precisely consistent snapshot of the data to the new shard. Once the data copy is complete, writing resumes and the accumulated updates are synchronized. This is a mechanism that trades brief business interruption for extremely high data consistency. Alternatively, a dual-write mechanism can be used, writing new data simultaneously to both the existing and new shards to ensure no data loss. The specific method chosen depends on the business's requirements for data consistency and service availability.

[0106] S402, Based on the candidate database sharding and table partitioning strategy and the risk warning, generate a routing strategy and a gray-scale switching strategy.

[0107] The routing strategy includes rules for determining data routing paths based on sharding keys, and the gray-scale switching strategy includes the order of gradually switching traffic proportions, the interfaces and metrics that need to be monitored after the switch, and the judgment conditions that need to be met before a full switch.

[0108] Specifically, the routing strategy includes rules for determining data routing paths based on sharding keys. This refers to a routing function that establishes a mapping relationship between sharding key values ​​and target shards based on the sharding keys and sharding rules defined in the candidate database sharding and table partitioning strategy. This is to determine how to calculate the target shard to which a business request should be routed based on the sharding key (e.g., user identifier) ​​carried in the request.

[0109] The gray-scale switching strategy includes: gradually switching the traffic proportion in sequence, setting the traffic switching proportion to a very small value. For example, first, 2% of the query traffic is switched to the new sharding structure. After successful verification, the proportion is gradually increased to 5%, then 50%, and finally a full switch is completed. The order of gradually switching the traffic proportion is based on the risk level predicted in the risk warning; the higher the risk, the smaller the initial proportion, the smaller the step interval, and the more cautious the switching process.

[0110] The interfaces and metrics that need to be monitored after the switch are related to the key performance indicators (such as P99 latency and throughput) listed in the policy verification checklist. These metrics are the objects that need to be monitored during the gray-scale switch.

[0111] The judgment conditions that need to be met before a full switchover are also derived from the thresholds set for each indicator in the policy verification checklist. For example, the next step of traffic amplification is allowed only when all monitored performance indicators consistently and stably exceed a predetermined duration (such as 30 minutes) at a certain traffic ratio, and the observed values ​​are all better than the threshold requirements.

[0112] S403, Based on the business topology and consistency requirements in the business data access pattern profile, generate consistency and transaction assurance guidelines.

[0113] The guarantee guidelines include requirements for maintaining the same database query path in the consistency chain, and suggestions for retaining the old database as the primary data source during the migration transition period for accounting scenarios.

[0114] Specifically, the requirement for maintaining the same database query path for strong consistency links means that during the migration process, for business operations that must guarantee strong consistency (such as payment transactions), the system needs to plan to ensure that the relevant data can still be operated within the same database shard before and after the migration (i.e., through local transactions, rather than cross-database transactions).

[0115] The recommendation to retain the old database as the primary data source during the migration transition period for accounting scenarios represents an operational suggestion for scenarios with high data consistency requirements. Specifically, before the migration is complete, it is recommended that such critical business scenarios continue to read data from the existing database shards to avoid accounting issues caused by delays or errors in synchronizing data from the new shards.

[0116] S404, Based on the aforementioned risk warning, generate a rollback contingency plan.

[0117] The rollback plan includes rollback triggering conditions, rollback step sequence, and data consistency checkpoints to be verified after rollback.

[0118] Specifically, the rollback trigger conditions are directly mapped from the risk type warned in the risk alert and the performance indicator thresholds set in the policy verification list. For example, the "potential query latency increase risk" warned in the risk alert is quantified here as a specific, monitorable threshold condition: "If the system's P99 quantile latency exceeds the preset threshold (such as 250 milliseconds) in the policy verification list during the gray-scale transition, a rollback will be triggered immediately."

[0119] The rollback step sequence indicates the execution sequence of the rollback operation. It can be the reverse process of the canary switch operation. For example: First, switch the traffic route back to the original shard; second, stop writing data to the new shard; third, perform data consistency verification.

[0120] Post-rollback data consistency checkpoints explicitly define which critical business data must be checked for consistency after the rollback operation is completed. For example, they verify whether the corresponding records for specific business identifiers are completely consistent in the original and new shards to ensure that the rollback operation itself did not introduce new data errors. In other words, a checkpoint is one or a set of critical business data records. These are typically identified by a unique business identifier, such as an order ID, a transaction serial number, or a user account ID. Each checkpoint requires a verification operation. The purpose of the verification is to confirm whether the state of the data is completely consistent in the source database (old shard) and the target database (new shard) after the rollback. This includes, but is not limited to: Data existence: Whether the data exists in both places. Field consistency: Whether the values ​​of all critical fields (such as amount, status, timestamp) are the same.

[0121] For example, suppose a "checkpoint" defined in the rollback plan is "the order record with order ID 202405200001". Then, after the rollback operation is completed, the database administrator needs to perform a check based on this checkpoint: query the record of this order ID in both the old and new shards, and confirm that the values ​​of its core fields such as order status, payment amount, and update time are completely consistent.

[0122] S405, integrate the data migration step planning, the routing strategy and canary switching strategy, the consistency and transaction guarantee guidelines and the rollback contingency plan to obtain an executable operation and maintenance manual.

[0123] The executable operation and maintenance manual is provided for database administrators (DBAs) to manually implement database sharding and table partitioning schemes.

[0124] Specifically, based on the actual operational time sequence and logical dependencies, the outputs of steps S401 to S404 are integrated into a structured document. For example, the document structure is clearly divided into sections such as "Preparation Phase," "Data Migration Phase," "Gradual Switching Phase," and "Rollback Plan." Each phase includes specific operation instructions, verification commands, and key decision points. The resulting executable operation manual enables database administrators to clearly and safely manually implement the entire database sharding and table partitioning solution without requiring automated system operations.

[0125] It should be noted that integrating the data migration step plan, the routing strategy and canary switchover strategy, the consistency and transaction guarantee guidelines, and the rollback contingency plan can be understood as a combination. This can be achieved by combining these elements, or by filling them into a document template. For example, the "data migration step plan" can be embedded in the "Data Migration Phase" section of the manual, and the "rollback contingency plan" can be embedded in the "Emergency Operations" section. Furthermore, the order in which these elements are integrated will be determined by the timeline and causal dependencies of the operations, resulting in their order of appearance and connection within the executable operation manual. For example, "data migration" must be completed before "canary rollout" can begin; while the preparation information for the "rollback plan" needs to be provided in advance, but its specific operational steps are arranged after the switchover process as a safeguard. This arrangement ensures the executability of the executable operation and maintenance manual.

[0126] S500: Based on the implementation effect of the executable operation and maintenance manual, obtain strategy implementation effect monitoring data.

[0127] Specifically, obtaining strategy implementation effect monitoring data may include the following steps: after the database administrator (DBA) completes the execution of the database sharding and table partitioning scheme according to the executable operation and maintenance manual, the key performance indicator data during runtime is collected to obtain strategy implementation effect monitoring data.

[0128] The key performance indicator data are operational metrics used to evaluate the implementation effect of the candidate database sharding and table partitioning strategy.

[0129] Specifically, the actual effects of the new database sharding and table partitioning strategy after its implementation in the production environment are objectively recorded, providing factual basis for subsequent effect evaluation and optimization. This is achieved by collecting key data that truly reflects changes in system performance before and after the strategy's implementation.

[0130] For example, this step can be triggered automatically or manually after the database administrator completes a series of operations such as deployment of the database sharding scheme, data migration, and canary deployment switching according to the executable operation and maintenance manual generated by the S400 series steps, and confirms that the new strategy has fully taken over production traffic.

[0131] From the production environment's monitoring system, collect runtime data that shares the same source as the key performance indicators defined in the strategy verification checklist generated in step S301. These indicators provide direct evidence for evaluating the effectiveness of the strategy and mainly include: database performance indicators such as table size, query latency (average latency, P95 / P99 quantile latency), requests per second, index hit rate, lock wait time, and the proportion and duration of cross-database transactions.

[0132] Application performance metrics: such as response time, throughput, and error rate of key business interfaces.

[0133] Resource utilization metrics: such as CPU utilization, memory usage, and disk I / O throughput of database servers.

[0134] Furthermore, to ensure the comprehensiveness and representativeness of the data, data collection continues for a complete business cycle (e.g., 24 hours) to cover peak and off-peak periods. This yields monitoring data on the effectiveness of strategy implementation. This is a time-series data set of the aforementioned key performance indicators over a period of time (e.g., 24 hours), reflecting the system status after the new strategy has been running stably.

[0135] It should be noted that key performance indicators (KPIs) can map to and serve the optimization goals set at the outset of the candidate database sharding strategy. For example, if the main goal of the solution is to resolve access hotspots and improve throughput, the KPIs will focus on the request rate of hotspot shards, the overall system queries per second (SPS) rate, and the CPU utilization of the database server. If the main goal of the solution is to reduce query latency and improve user experience, the KPIs will focus on the average response time of application-side interfaces, P95 and P99 quantile latency, etc. If the main goal of the solution is to optimize data distribution and reduce cross-shard operations, the KPIs will specifically monitor the proportion of cross-shard queries, the number and duration of cross-shard distributed transactions, etc.

[0136] These key performance indicators collectively form the quantitative evidence basis for evaluating the effectiveness of the strategy implementation. By comparing and analyzing the data of these indicators collected after the strategy implementation with the historical baseline data collected before the strategy implementation, it is possible to objectively determine whether the candidate database sharding and table partitioning strategy has achieved the expected goals and the specific performance improvement it has brought, thereby providing data support for subsequent strategy optimization or experience accumulation.

[0137] S600, extract the database sharding and table partitioning rule constraints from the monitoring data of the strategy implementation effect, and generate a strategy knowledge base.

[0138] In the subsequent step of obtaining the business data access pattern profile, the database sharding and table partitioning rules in the strategy knowledge base are used as one of the input conditions for generating the business data access pattern profile.

[0139] Specifically, generating a strategy knowledge base may include the following steps: S601, comparing and analyzing the strategy implementation effect monitoring data with the expected effect of the candidate database sharding and table partitioning strategy scheme, and generating a strategy implementation effect report.

[0140] Specifically, the actual monitoring data is compared with the expected results item by item. For example, the actual monitored cross-database query ratio is compared with the expected 5% threshold, and the actual measured P99 latency is compared with the expected 100 millisecond target. The analysis focuses on identifying three types of results: first, indicators that meet or exceed expectations; second, indicators that do not meet expectations but are acceptable; and third, indicators that do not meet expectations and cause new problems.

[0141] Based on the comparison results, the system generates a strategy implementation effectiveness report. This report not only lists the actual and expected values ​​of each key performance indicator but also performs attribution analysis on the differences. For example, the report states: "After using user_id as the sharding key, the latency of user-dimensional queries decreased by 60%, meeting expectations; however, because report queries could not hit the sharding key, a full shard scan was performed, resulting in a 200% increase in latency, failing to meet expectations."

[0142] S602, extract the database sharding and table partitioning rules and constraints from the strategy implementation effect report, and then deposit the extracted database sharding and table partitioning rules and constraints into the strategy knowledge base.

[0143] The strategy knowledge base is used as one of the input conditions when the subsequent steps of collecting database and business operation indicators, combined with business topology and consistency requirements, are performed to obtain a business data access pattern profile.

[0144] Specifically, the successful experiences and lessons learned from the strategy implementation results report are transformed into reusable database sharding and table partitioning rules.

[0145] For example, positive rules can be extracted from successful experiences: for instance, if a report shows that sharding by tenant_id (tenant ID) significantly improves query performance within a tenant, the extracted rule would be: "For business tables with strong tenant isolation characteristics in their access patterns, it is recommended to use tenant_id as the sharding key."; and constraints can be extracted from lessons learned from failures: for instance, if a report shows that the performance of report queries that cannot hit the sharding key deteriorates, the extracted constraint would be: "For operational and reporting businesses that require full table scans or complex condition queries, the tables they depend on should be sharded cautiously, or supplementary solutions such as read-write separation and heterogeneous indexes should be considered."

[0146] Next, the system will structure the extracted rules and constraints according to a unified format (e.g., rule type, applicable scenario, specific content, source basis) and store them in the strategy knowledge base. This strategy knowledge base is a dedicated best practice knowledge base for enterprise-specific database sharding and table partitioning.

[0147] Finally, the system will automatically query the strategy knowledge base the next time it initiates the S100 step (i.e., "collecting operational metrics of the database and business, and combining business topology and consistency requirements to obtain a business data access pattern profile") for a new business or database. Historical rules and constraints relevant to the current scenario from the strategy knowledge base will be used as important input conditions and injected into the generation process of the new profile, thereby ensuring that historical experience is inherited, avoiding repeating past mistakes, and achieving continuous self-optimization.

[0148] What's important is that when parsing business topology and consistency requirements, the system retrieves historical rules from the strategy knowledge base that match the current business characteristics, and transforms these sharding and partitioning rules into additional constraint attributes, injecting them into the business data access pattern profile being built. For example, when a "customer service query" scenario is identified, the system retrieves the rule "customer service queries must be routed by primary key" from the knowledge base, and then adds the constraint attribute "force single shard routing" to the profile for this scenario, making historical experience an inherent part of the profile, thereby directly affecting subsequent strategy generation.

[0149] Secondly, this application provides a dynamic data sharding and partitioning strategy generation system based on a large model, applied to the aforementioned dynamic data sharding and partitioning strategy generation method based on a large model. The system includes: a collection unit for collecting operational metrics of the database and business processes, and combining these metrics with business topology and consistency requirements to obtain a business data access pattern profile; an inference unit for obtaining candidate sharding and partitioning strategy schemes based on the business data access pattern profile using large language model inference technology; a generation unit for performing risk assessment on the candidate sharding and partitioning strategy schemes, and generating a strategy verification list and risk warnings; and a processing unit for processing the strategy verification list and risk warnings using migration path planning technology. The system employs a technique to obtain an executable operation and maintenance manual. The migration path planning technique includes migration steps, canary deployment strategies, transaction assurance guidelines, and rollback contingency plans. This executable operation and maintenance manual is for manual execution by database administrators (DBAs). An effect monitoring unit is used to obtain strategy implementation effect monitoring data based on the implementation effect of the executable operation and maintenance manual. A result unit is used to extract sharding and partitioning rule constraints from the strategy implementation effect monitoring data to generate a strategy knowledge base. In the subsequent step of obtaining a business data access pattern profile, the sharding and partitioning rule constraints in the strategy knowledge base are used as one of the input conditions for generating the business data access pattern profile.

[0150] Thirdly, this application provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.

[0151] Fourthly, this application provides a computer program that, when executed by a processor, implements the steps of the aforementioned method.

[0152] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0153] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0154] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating dynamic data sharding and table partitioning strategies based on a large model, characterized in that, include: Collect operational metrics of the database and business operations, and combine them with business topology and consistency requirements to obtain a profile of business data access patterns; Based on the business data access pattern profile, candidate database sharding and table partitioning strategies are obtained through large language model inference technology. A risk assessment is performed on the candidate database sharding and table partitioning strategies, and a strategy verification list and risk warnings are generated. Based on the strategy verification checklist and risk warnings, an executable operation manual is obtained through migration path planning technology. This migration path planning technology includes migration steps, canary deployment strategies, transaction assurance guidelines, and rollback contingency plans. The executable operation manual is for manual execution by database administrators (DBAs). Based on the implementation effect of the executable operation manual, strategy implementation effect monitoring data is obtained. From this monitoring data, sharding and partitioning rules are extracted to generate a strategy knowledge base. In the subsequent step of obtaining a business data access pattern profile, the sharding and partitioning rules in the strategy knowledge base are used as one of the input conditions for generating the business data access pattern profile.

2. The method for generating dynamic data sharding and table partitioning strategies based on large model-driven approaches as described in claim 1, characterized in that, The steps for collecting operational metrics of the database and services, combined with business topology and consistency requirements, to obtain a profile of business data access patterns include: collecting table structure metadata of the database; collecting database-side operational metrics to obtain a first set of metrics, wherein the first set of metrics includes table capacity, single table growth rate, frequency of inserts, deletes, and modifications, index hit rate, lock wait status, and cross-database transaction distribution; collecting application-side operational metrics of the database to obtain a second set of metrics, wherein the second set of metrics includes interface response time, P99 latency, typical query statements, and transaction call chain span; and analyzing the microservice interactions based on the first set of metrics and the second set of metrics. The system calls the data tables that each service depends on, generates a business topology relationship, and sets consistency requirements and constraints for the business topology relationship. It then integrates the table structure metadata, the first indicator set, the second indicator set, the business topology relationship, and the consistency requirements and constraints for the business topology relationship to generate a structured business data access pattern profile. This profile includes at least table structure information obtained from the table structure metadata, access pattern characteristics and data growth trends derived from analyzing the first and second indicator sets, and business priority information derived from analyzing the business topology relationship and consistency requirements and constraints.

3. The method for generating dynamic data sharding and table partitioning strategies based on large model-driven approaches as described in claim 1, characterized in that, The step of obtaining candidate database sharding and table partitioning strategies based on the business data access pattern profile and through large language model reasoning technology includes: encapsulating the structured information in the business data access pattern profile into standardized prompts; inputting the standardized prompts into the large language model, and generating candidate database sharding and table partitioning strategies through semantic analysis, wherein the candidate database sharding and table partitioning strategies include recommended sharding types, reasons for selecting sharding keys, suggestions for hot data isolation, and consistent access path planning.

4. The method for generating dynamic data sharding and table partitioning strategies based on large model-driven approaches as described in claim 1, characterized in that, The step of performing a risk assessment on the candidate database sharding and table partitioning strategy and generating a strategy verification list and risk warnings includes: generating a strategy verification list based on the candidate database sharding and table partitioning strategy, wherein the strategy verification list includes typical query and transaction scenarios that need to be replayed, key performance indicators that need to be observed, and their thresholds; and generating risk warnings based on the candidate database sharding and table partitioning strategy, wherein the risk warnings include predictions of potential performance bottlenecks and consistency risks.

5. The method for generating dynamic data sharding and table partitioning strategies based on large model-driven approaches as described in claim 1, characterized in that, The steps for obtaining an executable operation and maintenance manual based on the strategy verification checklist and risk warnings, using migration path planning technology, include: generating a data migration step plan based on the candidate database sharding and table partitioning strategy schemes, the strategy verification checklist, and the risk warnings. This data migration step plan includes: recommending priority business lines or channels for migration, the time window or data range for batch migration, and the method for handling newly written data during migration; and generating routing and canary switchover strategies based on the candidate database sharding and table partitioning strategy schemes and the risk warnings. The routing strategy includes rules for determining data routing paths based on sharding keys, and the canary switchover strategy includes the order of gradually switching traffic proportions, the interfaces and metrics that need to be monitored after the switchover, and the requirements that need to be met before a full switchover. The judgment conditions are as follows: Based on the business topology and consistency requirements in the business data access pattern profile, a consistency and transaction guarantee guide is generated. This guide includes requirements for maintaining the same database query path for consistency links, and suggestions for retaining the old database as the primary data source during the migration transition period for accounting scenarios. Based on the risk warnings, a rollback plan is generated. This plan includes rollback triggering conditions, rollback step sequence, and data consistency checkpoints after rollback. The data migration step plan, routing strategy and canary switching strategy, consistency and transaction guarantee guide, and rollback plan are integrated to obtain an executable operation and maintenance manual. This manual is used by database administrators (DBAs) to manually implement database sharding and table partitioning schemes.

6. The method for generating dynamic data sharding and table partitioning strategies based on large model-driven approaches as described in claim 1, characterized in that, The step of obtaining strategy implementation effect monitoring data based on the implementation effect of the executable operation and maintenance manual includes: after the database administrator (DBA) completes the execution of the database sharding and table partitioning scheme according to the executable operation and maintenance manual, the DBA collects key performance indicator data during the operation to obtain strategy implementation effect monitoring data, wherein the key performance indicator data are operational indicators used to evaluate the implementation effect of the candidate database sharding and table partitioning strategy scheme.

7. The method for generating dynamic data sharding and table partitioning strategies based on large model-driven approaches as described in claim 1, characterized in that, The step of extracting sharding and table partitioning rules and constraints from the strategy implementation effect monitoring data to generate a strategy knowledge base includes: comparing and analyzing the strategy implementation effect monitoring data with the expected effects of the candidate sharding and table partitioning strategy schemes to generate a strategy implementation effect report; extracting sharding and table partitioning rules and constraints from the strategy implementation effect report, and storing the extracted sharding and table partitioning rules and constraints in the strategy knowledge base. The strategy knowledge base is used as one of the input conditions in the subsequent step of executing the steps of collecting database and business operation indicators, combined with business topology and consistency requirements, to obtain a business data access pattern profile.

8. A dynamic data sharding and table partitioning strategy generation system based on a large model, characterized in that, The method for generating dynamic data sharding and partitioning strategies based on a large model, applicable to any one of claims 1-7, comprises: a collection unit for collecting operational metrics of the database and business operations, and obtaining a business data access pattern profile by combining business topology and consistency requirements; an inference unit for obtaining candidate sharding and partitioning strategy schemes based on the business data access pattern profile using large language model inference technology; a generation unit for performing risk assessment on the candidate sharding and partitioning strategy schemes, and generating a strategy verification list and risk warnings; and a processing unit for obtaining executable operation and maintenance operations based on the strategy verification list and risk warnings using migration path planning technology. The manual includes migration path planning technology comprising migration steps, canary deployment strategies, transaction protection guidelines, and rollback contingency plans. The executable operation manual is for manual execution by database administrators (DBAs). An effect monitoring unit is used to obtain strategy implementation effect monitoring data based on the implementation effect of the executable operation manual. A result unit is used to extract sharding and partitioning rule constraints from the strategy implementation effect monitoring data to generate a strategy knowledge base. In the subsequent step of obtaining a business data access pattern profile, the sharding and partitioning rule constraints in the strategy knowledge base are used as one of the input conditions for generating the business data access pattern profile.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images