Database expansion methods, devices, equipment, storage media, and program products

By obtaining the number of extended data shards, creating an extended database based on the database's data size and load information, and performing migration based on the data sharding field, the problems of low automation and uneven resource utilization in existing technologies are solved, achieving efficient and secure database expansion.

CN123115280APending Publication Date: 2026-11-17CHINA UNIONPAY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611241416.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, the database expansion process relies on manual configuration of migration task details, resulting in low automation, low expansion efficiency, complex data migration, uneven resource utilization, and the absence of an automatic recycling mechanism, leading to idle resources and low data reliability.

Method used

By obtaining the number of extended data shards, an extended database is created using the database's data size and operating load information. Data migration is then performed based on the data shard field, and combined with consistency checks and traffic switching, an automated database expansion process is achieved.

Benefits of technology

It improves the automation of database expansion, optimizes resource allocation, ensures the accuracy and consistency of data migration, reduces manual operations, and enhances expansion efficiency and security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115280A_ABST
    Figure CN123115280A_ABST
Patent Text Reader

Abstract

This application discloses a database expansion method, apparatus, device, storage medium, and program product, belonging to the field of artificial intelligence technology. The method obtains the number of extended data shards used to perform database expansion; based on the database's data size information, operational access load information, and shard deployment configuration information of the source data shards in the database, an extended database is created, the number of extended data shards included in the extended database being the number of extended data shards; according to the data shard field of the stored data corresponding to the source data shards, the replica data of the stored data is migrated to the extended database. In this way, by simplifying user operations, reusing the shard deployment configuration information of the source data shards to accelerate the creation of the extended database, and completing the replica data migration based on the data shard field, the automation level of the database expansion process is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a database expansion method, apparatus, device, storage medium and program product. Background Technology

[0002] As the scale of service data continues to grow, the storage capacity of the database may no longer meet service demands, necessitating database expansion. The entire database is split into multiple data subsets, each stored in a data shard within the database. Currently, database expansion can be achieved by creating extended data shards larger than the number of source shards and then re-splitting and migrating the data from the source shards to the extended data shards according to manually entered data mapping information. However, this method, relying on manual configuration of data mapping information, severely limits the automation of the database expansion process. Summary of the Invention

[0003] This application provides a database expansion method, apparatus, device, storage medium, and program product, which can solve the problem of low automation in database expansion.

[0004] In a first aspect, embodiments of this application provide a database expansion method, the method comprising: Get the number of extended data shards used to perform database expansion; Based on the database's data size information, runtime access load information, and the sharding deployment configuration information of the source data shards in the database, an extended database is created. The number of extended data shards included in the extended database is the number of extended data shards. Based on the data sharding field of the storage data corresponding to the source data shard, the copy data of the storage data is migrated to the extended database.

[0005] Secondly, embodiments of this application provide a database expansion device, the device comprising: The acquisition module is used to obtain the number of extended data shards used to perform database expansion; The creation module is used to create an extended database based on the database's data size information, runtime access load information, and the sharding deployment configuration information of the source data shards in the database. The number of extended data shards included in the extended database is the number of extended data shards. The migration module is used to migrate the copy data of the stored data to the extended database based on the data sharding field of the stored data corresponding to the source data shard.

[0006] Thirdly, embodiments of this application provide a computer device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the database expansion method as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the first aspect.

[0008] Fifthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the first aspect.

[0009] The database expansion method, apparatus, device, storage medium, and program product of this application embodiment obtain the number of extended data shards used to perform database expansion; based on the database's data size information, operational access load information, and shard deployment configuration information of the source data shards in the database, an extended database is created, and the number of extended data shards included in the extended database is the number of extended data shards; according to the data shard field of the stored data corresponding to the source data shards, the copy data of the stored data is migrated to the extended database. In this way, users do not need to manually generate a complex migration task list; they only need to input the number of extended data shards to start the database expansion process. When creating the extended database, the efficiency of creating the extended database is improved by reusing the shard deployment configuration information of the source data shards. At the same time, by comprehensively considering the database's data size information and operational access load information, dynamic optimization of database resources is achieved, effectively avoiding resource idleness and waste. Subsequently, by using a method based on the data shard field of the migrated data itself to migrate the copy data of the migrated data, the copy data can be accurately routed to the target extended data shard, ensuring the accuracy of data migration. In this way, by simplifying user operations, reusing the sharding deployment configuration information of the source data shards to accelerate the creation of extended databases, and completing the migration of replica data based on the data sharding field, the automation level of the database expansion process is improved. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a database expansion system provided in an embodiment of this application; Figure 2A flowchart illustrating a database expansion method provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a database expansion device provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0013] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0014] The acquisition, storage, use, and processing of data (including but not limited to features and information mentioned in this document) in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0015] As the scale of service data continues to grow, service organizations are increasingly demanding more storage capacity for their databases. In related technologies, database expansion can be achieved through the following methods: First, a migration task detail list needs to be manually generated based on the current distribution of source data shards in the database and the expected expansion target. Then, the configuration center generates a routing mapping table based on the migration task detail list, and uses a shard-by-shard serial migration method to sequentially perform data shard locking, data migration, and data unlocking operations on each source data shard in the database.

[0016] However, the aforementioned technologies have the following problems, as detailed below.

[0017] The reliance on manual configuration of migration task details for initiating database expansion severely restricts the automation level of the expansion process. Data migration during expansion uses a sequential, shard-by-shard migration method, resulting in a complex process and low efficiency. Data shard blocking during expansion affects the service provision of corresponding source data shards. The lack of an automatic reclamation mechanism for source data shards after expansion easily leads to resource idleness and database maintenance risks. Furthermore, the absence of an independent consistency verification step for migrated data after expansion results in low data reliability.

[0018] It is evident that the aforementioned technologies cannot meet the database expansion requirements of seamless business operations, high security and reliability, and high automation in database expansion scenarios.

[0019] To address the aforementioned shortcomings, this application proposes a database expansion method. This method improves the automation of the database expansion process by initiating the expansion process with an increased number of data shards. By comprehensively considering the database's data size, operational access load, and the shard deployment configuration information of the source data shards, it can allocate appropriate resources to the new expanded database, avoiding the uneven resource allocation problem that may exist in related technologies. Furthermore, it improves database expansion efficiency by reusing the shard deployment configuration information of the source data shards. Finally, it ensures accurate consistency throughout the data expansion process through consistency checks before and after data migration.

[0020] The following will be combined with the appendix Figures 1 to 4 The database expansion methods, apparatus, devices, storage media, and program products described in this application are described in detail. It should be noted that these embodiments are not intended to limit the scope of this application.

[0021] The following is combined Figure 1 The database expansion system provided in the embodiments of this application will be described in detail.

[0022] Figure 1 This is a schematic diagram of a database expansion system provided in an embodiment of this application.

[0023] like Figure 1As shown, the database expansion system 10 in this embodiment is deployed on a container-based database cloud platform. The database cloud platform provides various types of database services. In this embodiment, a database refers to any system with data storage capabilities, including but not limited to: relational databases, in-memory databases, non-relational databases, and message queues. Relational databases may include, but are not limited to: the open-source distributed relational database TiDB, the enterprise-level distributed relational database OceanBase, and the distributed relational database UPDRDB based on open-source MySQL; in-memory databases refer to databases that primarily use memory as their storage medium, such as a remote dictionary server (Redis).

[0024] Database cloud platforms utilize container orchestration engines such as Kubernetes or container orchestration tools such as Docker Swarm for container deployment and management. Each database instance within the database cloud platform runs as an independent container on a host machine managed by the platform, and is allocated dedicated computing resources such as a CPU, memory, and network resources such as a dedicated Internet Protocol (IP) address, as well as independent storage space such as tablespaces and log spaces. Tablespaces are used to store data, and log spaces are used to store database logs.

[0025] The database expansion system 10 in this application embodiment may include a sharding preparation subsystem 101, a data migration subsystem 102, a data consistency verification subsystem 103, a traffic switching subsystem 104, and a resource reclamation subsystem 105.

[0026] The database expansion method provided in the embodiments of this application will be described in detail below.

[0027] The sharding preparation subsystem 101 is the data entry point for the database expansion system 10 provided in this application embodiment. It is used to identify the number of extended data shards for performing database expansion from the database expansion request, and to create an extended database that matches the number of extended data shards by cloning and reusing the sharding deployment configuration information of the database.

[0028] The sharding preparation subsystem 101 is used to extract the database identifier and database type carried in the database expansion request issued by the database cloud platform, and obtain the database's running status and the task execution status of the data operation and maintenance tasks by calling the database's running status interface and task status interface. The data operation and maintenance tasks include at least one of the following: data backup task, data recovery task, data migration task, data synchronization task, and data verification task; the database's running status includes any one of the following: normal running status and abnormal running status; the task execution status includes any one of the following: no data operation and maintenance task execution status and data operation and maintenance task execution status. For example, if the database identifier is updrdb002, the database type identifier is UPDRDB, the running status is normal running status, and the data operation and maintenance task execution status is no data operation and maintenance task execution status.

[0029] The sharding preparation subsystem 101 is also used to, when the database is in normal operation and the data operation and maintenance task is in a state of no data operation and maintenance task execution, call the cloud platform's resource pool status interface to obtain the amount of idle resources on the cloud platform; and, extract the number of extended data shards used to perform database expansion from the database expansion request, and verify whether the number of extended data shards for database expansion is within a preset compliant quantity range, to obtain the database expansion request verification result and the amount of idle resources. For example, if the number of extended data shards is 4, and the preset compliant quantity range is 2-10, the corresponding database expansion request verification result is "verification passed"; the amount of idle resources includes 10 CPU resources of 8 cores, 10 memory resources of 32GB, and 10 disk resources of 500GB.

[0030] The sharding preparation subsystem 101 is also used to, when the number of extended data shards is within a preset compliant range and the idle resources of the cloud platform meet the sharding extension resource conditions corresponding to the database expansion request, call the database cloud platform metadata interface to collect the sharding deployment configuration information of the source data shards in the database, and create an extended database based on the database's data scale information, running access load information, and the sharding deployment configuration information of the source data shards in the database. The sharding deployment configuration information includes sharding metadata, proxy component configuration information, and switch management component configuration information. The sharding metadata is information required for deploying and scheduling source data shards and executing data partitioning, and may include, but is not limited to: database version, scale such as CPU and memory, data disk size, log disk size, source data sharding routing rules, and source data sharding cluster architecture information; the proxy component configuration information includes the routing configuration information of the proxy component, specifically the routing configuration information of the database proxy component Proxy; and the switch management component configuration information includes the routing configuration information of the switch management component, specifically the routing configuration information of the switch management component Switchmanager. The Proxy is used to accurately route data access requests to the corresponding data shards according to the sharding routing rules, while the Switchmanager is used to monitor the running status of the data shards. For example, sharding deployment configuration information may include, but is not limited to: version V1.8.10, 8 CPU cores, 32GB memory, 500GB disk, source data sharding routing rules, Proxy routing configuration information, and Switchmanager routing configuration information; the extended data shard creation result is "created successfully"; the component reuse result is "Proxy and Switchmanager reuse completed," and the extended data shard identifier is updrdb_xx.

[0031] The sharding preparation subsystem 101 is also used to perform the following steps when the extended data sharding creation result is successful, the data routing link configuration corresponding to the Proxy is normal, and the management link configuration corresponding to the Switchmanager is normal: Start the extended data sharding service using the extended data sharding initialization configuration script and check whether the running status of each extended data sharding service is normal to obtain the extended data sharding running status; perform consistency verification on the sharding-related configuration information of the extended data shards and the sharding-related configuration information of the source data shards to obtain the consistency verification result, and generate a consistency verification report if the consistency verification result is successful, thereby triggering the execution of the database cloud platform to issue a data migration request to the data migration subsystem 102. The sharding-related configuration information may include, but is not limited to: database configuration parameters, data table structure information, user permission information, and resource configuration type. For example, the extended data sharding running status is normal; the consistency verification report is successful; the extended data sharding preparation completion result is successful; the extended data sharding node identifier is updrdb_xx; the database type is UPDRDB; and whether consistency verification is triggered is yes.

[0032] The data migration subsystem 102 is used to match the database export engine according to the database type of the database, export the copy data of the data to be migrated in the source data shard based on the database export engine, import the copy data of the data to be migrated into the extended data shard in the extended database based on the extended data shard routing rules, and push the data migration task execution status to the data consistency verification subsystem 103.

[0033] Specifically, the data migration subsystem 102 extracts the database type identifier from the data migration request issued by the database cloud platform. After successful extraction of the database type identifier, it parses the data migration request to obtain parsed parameters. These parsed parameters may include, but are not limited to: database identifier, number of source data shards, number of extended data shards, source data shard connection information, and extended data shard connection information. Here, the source data shard connection information includes the source data shard IP, database port, access account, account password, and default database name; the extended data shard connection information includes the extended data shard IP, extended database port, access account, account password, and default database name. If parameter parsing is successful, a pre-migration environment verification step is performed on the extended database, and the parameter parsing results and pre-migration environment verification results are output. The pre-migration environment verification step may include, but is not limited to: network connectivity check, database account permission check, and disk space check. Here, the network connectivity check is used to verify whether the networks between all source data shards and all extended data shards are interconnected; the database account permission check is used to verify whether both the database account and the extended database account have read and write permissions; and the disk space check is used to verify whether the remaining space in the extended database is sufficient to meet the data storage requirements for migration. For example, the migration pre-migration environment verification result obtained by the data migration subsystem 102 is "verification passed".

[0034] The data migration subsystem 102, after the pre-migration environment verification passes, also calls a data export engine matching the database type from the data export engine central system. Based on the data export engine and the original data shard connection information, it performs data export on the stored data in each source data shard in the database, generating a Structured Query Language (SQL) file corresponding to each source data shard. Each SQL file includes a copy of the stored data corresponding to the source data shard. For example, using a MySQL database backup program compatible with mysqldump, the original shard 0 and original shard 1 are exported respectively. Thus, the data migration subsystem 102 obtains: the data export results include shard0.sql and shard1.sql, and the export status of the copy data of each source data shard is "successfully exported".

[0035] The data migration subsystem 102 is also used to start the incremental parsing component during the data export process of the stored data in each source data shard in the database. The incremental parsing component reads the position information corresponding to the export task of each source data shard to locate the starting position of incremental data extraction of each source data shard. The incremental parsing component identifies data change events of the insertion, update and deletion types in the log, performs format standardization processing on the incremental events to obtain the shard data incremental log entries, and pushes the shard data incremental log entries to the incremental queue for temporary storage.

[0036] The data migration subsystem 102 is also used to, after completing the incremental data capture of each source data shard, call a data import tool such as a MySQL import tool to import the SQL files corresponding to each source data shard in the database and the incremental data corresponding to each source data shard in the incremental queue. Data import is performed according to the data shard fields, such as the primary key identifier (ID), and the extended data shard routing rules calculated based on the source data shard routing rules of the database. The import order is as follows: first, the copy data of the stored data in the SQL file corresponding to each source data shard is imported into the extended data shard of the extended database; then, the incremental data corresponding to each source data shard is imported into the extended data shard of the extended database. For example, the source data shard routing rule is to perform a modulo operation on the primary key ID to determine the source data shard, such as ID%2; correspondingly, the extended data shard routing rule can be ID%4. For example, the data migration subsystem 102 can obtain the following data import results: Extended data shard 0: Success, Extended data shard 1: Success, Extended data shard 2: Success, Extended data shard 3: Success.

[0037] The data migration subsystem 102 is also used to close the incremental parsing component, clean up the SQL file and the cache file in the incremental queue, mark the current data migration task execution status as completed, and push the following execution result to the data consistency verification subsystem 103 when the data import result is that the copy data of each source data shard and the incremental data are successfully imported.

[0038] The data consistency verification subsystem 103 is used to perform data migration integrity verification on the extended database when the data migration task pushed by the data migration subsystem 102 is in the completed state, obtain the integrity verification result, and output the integrity verification result to the traffic switching subsystem 104.

[0039] Specifically, the data consistency verification subsystem 103, upon receiving a data migration task pushed by the data migration subsystem 102 with a completed execution status, generates and initiates a verification task based on the source data shard information and the extended data shard information. For example, the database is updrdb002 with 2 source data shards; the extended database has 4 extended data shards. The data consistency verification subsystem 103 can output the following execution result: Verification task started, ID: check_20260316_001.

[0040] The data consistency verification subsystem 103 is also used to compare the database configuration parameters of the migrated extended data shard with those of the source data shard when the verification task is started, and output the fifth verification result. For example, the database configuration parameters may include, but are not limited to: database character set, database instance service listening port, account permission set, and data table storage engine.

[0041] The data consistency verification subsystem 103 is further configured to, if the fifth verification result is successful, compare the total amount of data in the migrated data in the extended database with the total amount of data in the corresponding migrated data in the extended database, and output a sixth verification result. For example, the total amount of data may include at least one of the following: total number of data rows and total data size. For instance, if the total number of rows in the two source data shards of the database is 3.99 million, and the total number of rows in the four extended data shards of the extended database is also 3.99 million, then the sixth verification result is determined to be successful.

[0042] The data consistency verification subsystem 103 is also used to verify whether the first sample data belongs to the corresponding reference extended data shard according to the extended data sharding routing rules of the extended database, and output the seventh verification result if the sixth verification result is successful. For example, the source data sharding routing rule is ID%2, the extended data sharding routing rule is ID%4, and the hash modulus amplification factor is 4 / 2=2; the data of source data shard 0 should be routed to target data extended data shards 0 and 2, and the data of source data shard 1 should be routed to target data extended data shards 1 and 3; random sample data is selected for routing matching verification, and if all sample data routing matches correctly, the seventh verification result is determined to be successful.

[0043] The data consistency verification subsystem 103 is also used to compare the data content of the second sample data in the extended database and the third sample data in the database, if the seventh verification result is successful, and output the eighth verification result, indicating that the data sharding fields of the second and third sample data are the same. For example, a specified number of third sample data are randomly selected from the database, and the corresponding second sample data is searched in the extended database according to the data sharding fields of the third sample data. Then, the data content of the second and third sample data is compared to check for missing, incorrect, or duplicate data. If no abnormalities are found, the following result is output: Data content verification: Passed.

[0044] The data consistency verification subsystem 103 is also used to summarize the fifth, sixth, seventh, and eighth verification results if the eighth verification result is successful, and to push a consistency verification success report to the traffic switching subsystem 104 if the total verification result is a full success. For example, the consistency verification success report may include information such as consistency verification being successful and traffic switching being permitted.

[0045] The traffic switching subsystem 104 is used to route data access requests for the source data shard to the extended data shard in the extended database by modifying the routing configuration information of the proxy component when it receives a consistency verification pass report pushed by the data consistency verification subsystem 103.

[0046] Specifically, the traffic switching subsystem 104, upon receiving a consistency verification pass report pushed by the data consistency verification subsystem 103, initiates a switching task and outputs the startup result based on the source data shard list, the extended data shard list, and the proxy component cluster status information. For example, the source data shards are: updrdb002_0 and updrdb002_1; the extended data shards are: 0-3; the generated switching ID is: switch_20260316_001; correspondingly, the startup result could be: the switching task has been started, ID: switch_20260316_001.

[0047] The traffic switching subsystem 104 is also used to generate proxy component health check results and configuration preloading results based on the proxy component cluster status and extended data shard routing rules, after confirming that the switching task has been started. Specifically, the traffic switching subsystem 104 generates routing configuration information adapted to the extended data shards according to the extended data shard routing rules; at the same time, based on the proxy component cluster status information, it checks the running and living status of all proxy components, whether the data routing links with the extended data shards are connected, and whether there are any conflicts in the routing configuration information. For example, corresponding routing configuration information is generated for the four extended data shards and distributed to each proxy component to complete the preloading. When the proxy component cluster status information indicates that three proxy components are online, the data links with each data shard are connected, and there are no conflicts in the routing configuration information, it is determined that the proxy component health check has passed and the configuration preloading is complete.

[0048] The traffic switching subsystem 104 is also used to update the effective routing rules by calling the proxy component's routing update interface after the proxy component's health check has passed and the configuration preloading has been completed. Based on the updated effective routing rules, it routes data access requests for a preset proportion of the source data shards to the extended data shards in the extended database, and detects the service operation indicators of the extended data shards within a preset time window to obtain the gray-scale switching operation status. For example, 50% of the service traffic is routed to 4 extended data shards; after continuous monitoring for 30 seconds, if the average response time of the extended data shards is less than or equal to 20ms, the number of connections is within a preset normal threshold range, and no error logs appear, it is determined that the gray-scale switching is in a normal state and the extended data shards are operating stably.

[0049] The traffic switching subsystem 104 is also used to, when the gray-scale switching is in a normal state and the extended data shards are running stably, call the routing update interface of the proxy component to switch the effective routing rules of the proxy component from the source data shard routing rules of the database to the extended data shard routing rules of the extended database. This is to route all data access requests for the source data shards to the extended data shards in the extended database; and to disable data write permissions for the source data shards, configuring only read-only access permissions. In this way, by only updating the effective routing rules of the proxy component, 100% of the data access requests can be routed to the four extended data shards, confirming that the full traffic switching is complete and the full data access requests are successfully distributed.

[0050] The traffic switching subsystem 104 is also used to detect the service operation indicators of the extended data shard within a preset observation time window after confirming that the full traffic switching is completed and the full data access request is successfully distributed, in order to obtain service switching determination information. For example, if the service operation indicators of the extended data shard within a preset observation time window, such as 5 minutes, are detected, and the average response time of the extended data shard is less than or equal to 20ms, the number of connections is within a preset normal threshold range, and no error logs appear, then the service switching determination information is determined to be that the service operation is stable and the full traffic switching has been confirmed to be effective.

[0051] The state switching subsystem 104 is also used to issue a source data fragment resource reclamation instruction to the resource reclamation subsystem 105 when the service switching judgment information determines that the service operation is stable and the full traffic switching is confirmed to be effective; and to mark the switching task as completed and generate a complete switching log.

[0052] The resource recycling subsystem 105 is also used to perform offline isolation operation, permission revocation operation, instance process termination operation, resource release operation and metadata cleanup operation on the source data fragment in sequence when it receives the source data fragment resource recycling instruction issued by the traffic switching subsystem 104.

[0053] Specifically, the resource recycling subsystem 105, upon receiving a source data fragment resource recycling instruction from the traffic switching subsystem 104, initiates a resource recycling task based on the source data fragment list and cloud platform resource information, and outputs the startup result. For example, the source data fragments are: updrdb002_0 and updrdb002_1; the generated recycling ID is: recycle_20260316_001; correspondingly, the startup result is: Resource recycling task has been started, ID: recycle_20260316_001.

[0054] The resource recycling subsystem 105 is also used to, when the resource recycling task has been started, call the routing management interface of the proxy component to delete the source data fragment routing rules related to the source data fragment in the proxy component; and call the connection management tool to disconnect the data transmission session for the source data fragment, close the data access port of the source data fragment, and determine that the source data fragment has been completely isolated.

[0055] The resource recycling subsystem 105 is also used to, when the source data shards have been completely isolated, call the permission management interface to revoke the authorized account's operation permissions for the source data shards, and switch the working mode of the source data shards from read-write mode to forced read-only mode, ensuring that all operation permissions for the source data shards are revoked and the read-only lock on the source data shards is effective. For example, this may include revoking the operation permissions of all accounts associated with updrdb002_0 and updrdb002_1, and locking updrdb002_0 and updrdb002_1 to read-only mode.

[0056] The resource recycling subsystem is also used to obtain the running status of the source data shard through the instance status check interface when all operation permissions of the source data shard have been revoked and the read-only lock of the source data shard is in effect. If it is confirmed that there are no background tasks being executed, no data writing operations being executed, and no log data to be flushed in the log buffer, the source data shard instance is determined to have been safely shut down.

[0057] The resource reclamation subsystem 105 is also used to, when the source data shard instance has been safely shut down, call the cloud platform resource management application programming interface (API) to batch release the computing resources, storage resources, and network resources occupied by the source data shards, and determine that the resource release operation of the source data shards is complete. The computing resources may include CPU and memory resources; the storage resources may include business data disk resources and backup disk resources; and the network resources include virtual IPs and external service port resources. For example, it batch releases the CPU, memory, and disk resources corresponding to updrdb002_0 and updrdb002_1, and reclaims the virtual IPs and external service port resources bound to updrdb002_0 and updrdb002_1.

[0058] The resource recycling subsystem 105 is also used to call the metadata management interface to clean up the fragment metadata, configuration information, and monitoring entries of the source data fragments stored on the cloud platform after the resource release operation of the source data fragments is completed; and to generate a standardized fragment recycling report based on the full-process operation log and archive and store it uniformly. The standardized fragment recycling report includes, but is not limited to: recycling object, recycling time, and recycling execution result.

[0059] It should be noted that all interface calls involved in the embodiments of this application must be verified for access control based on preset security credentials, and the corresponding data operations or management operations are only allowed after the verification is passed.

[0060] Based on this, the database expansion system provided in this application embodiment obtains the number of extended data shards for performing database expansion through the sharding preparation subsystem 101, and creates an extended database matching the number of extended data shards by cloning and reusing the database's sharding deployment configuration information. This eliminates the need for manual configuration of any underlying parameters, effectively improving the creation efficiency of the extended database and reducing resource consumption, thus achieving lightweight sharding expansion. The data migration subsystem 102 determines the database export engine that matches the database type, exports a copy of the data to be migrated from the source data shards based on the database export engine, and imports the copy of the data to be migrated into the extended data shards in the extended database based on the extended data shard routing rules. It then pushes the data migration task execution status to the data consistency verification subsystem 103. When the data migration task execution status pushed by the data migration subsystem 102 is "completed," the data consistency verification subsystem 103 performs a data migration integrity verification on the extended database, obtains the integrity verification result, and outputs the integrity verification result to the traffic switching subsystem 104. Upon receiving a consistency verification pass report pushed by the data consistency verification subsystem 103, the traffic switching subsystem 104 dynamically modifies the routing configuration information of the proxy component to route data access requests for the source data shard to the extended data shard in the extended database. Upon receiving a source data shard resource reclamation instruction from the traffic switching subsystem 104, the resource reclamation subsystem 105 sequentially performs offline isolation, permission revocation, instance process termination, resource release, and metadata cleanup operations on the source data shard. Thus, the database expansion system 10 provided in this embodiment, by executing the database expansion method provided in this embodiment, can achieve lightweight horizontal sharding expansion, ensure data consistency throughout the entire data expansion process, reduce manual input, and improve the level of automation across the entire database expansion chain, fully adapting to the business needs for seamless, secure, and highly automated database expansion.

[0061] It should be noted that the database expansion system provided in this application embodiment is applicable to service scenarios such as Internet business, massive data storage, and high-concurrency read and write services, and is used to solve the needs of database expansion, business traffic migration, and the iteration and decommissioning of old database instances.

[0062] Based on the database expansion system 10 mentioned above, this application embodiment also provides a system applicable to, for example... Figure 1 The database expansion method of the database expansion system 10 shown can be specifically combined with... Figure 2 Please provide a detailed explanation.

[0063] Figure 2 This is a flowchart of a database expansion method provided in an embodiment of this application.

[0064] like Figure 2 As shown, this database expansion method can be applied to, for example... Figure 1 The database expansion system shown may include steps 210 to 230.

[0065] Step 210: Obtain the number of extended data shards used to perform database expansion; Step 220: Create an extended database based on the database's data size information, running access load information, and shard deployment configuration information of the source data shards in the database. The number of extended data shards included in the extended database is the number of extended data shards; Step 230: Migrate the replica data of the stored data to the extended database based on the data shard field of the stored data corresponding to the source data shards.

[0066] Based on this, taking a distributed database used to store user order data as an example, the above steps 210 to 230 will be explained by way of example.

[0067] The distributed database comprises two source data shards: source data shard 0 and source data shard 1. As business volume increases, the distributed database needs to be expanded. The data expansion system receives a database expansion request from the cloud platform and, through the shard preparation subsystem 101, identifies the number of extended data shards (4) input from the database expansion request to perform the database expansion. The shard preparation subsystem 101 creates the extended database based on the distributed database's data size information, operational access load information, and the shard deployment configuration information of the source data shards in the database. The data size information may include the current total data volume, daily new data volume, and historical data growth trend; the operational access load information may include the number of data access requests, concurrent connections, CPU utilization, and memory utilization that the database handles within a statistical period; the shard deployment configuration information may include the database version, size (e.g., Central Processing Unit (CPU) and memory), data disk size, log disk size, source data shard routing rules, and source data shard cluster architecture information. Based on data scale information, runtime access load information, and the sharding deployment configuration information of the source data shards in the database, the sharding preparation subsystem 101 allocates resources for the four extended data shards to be created. For example, the sharding preparation subsystem 101 can allocate resources for each new extended data shard according to the data scale information, runtime access load information, and sharding deployment configuration information of the source data shards, and select four idle virtual machine instances for deployment on the cloud platform. These four newly deployed extended data shards constitute the extended database. Next, the data migration subsystem 102 migrates the replica data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shards. For example, a sharding route identifier is obtained by performing a modulo operation on the primary key ID of each order data, such as the order ID. The data migration subsystem 102 starts a data migration process. This data migration process first reads the replica data of the order data from the source data shards. For each replica data of an order, the data migration subsystem 102 recalculates the sharding route identifier according to the sharding routing rules corresponding to the extended data shards. For example, consider four extended data shards: Extended Data Shard 0, Extended Data Shard 1, Extended Data Shard 2, and Extended Data Shard 3. Order data with order ID 3 originally had its shard routing identifier pointing to source data shard 0. After expansion, since the number of extended data shards exceeds the number of source data shards, the new shard routing identifier points to extended data shard 2. The data migration subsystem 102, based on this new shard routing identifier, writes a copy of the order data to extended data shard 2 in the extended database. In this way, copies of all stored data in the source data shards are replicated to the new extended database, thus completing the database expansion process.

[0068] Therefore, by obtaining the number of extended data shards used to perform database expansion; based on the database's data size, operational access load, and the shard deployment configuration information of the source data shards in the database, an extended database is created, and the number of extended data shards included in the extended database is the number of extended data shards; based on the data shard field of the stored data corresponding to the source data shards, the replica data of the stored data is migrated to the extended database. In this way, users do not need to manually generate complex migration task lists; they only need to input the number of extended data shards to start the database expansion process. When creating the extended database, the efficiency of creating the extended database is improved by reusing the shard deployment configuration information of the source data shards. At the same time, by comprehensively considering the database's data size and operational access load information, dynamic optimization of database resources is achieved, effectively avoiding resource idleness and waste. Subsequently, by using a method based on the data shard field of the migrated data itself to migrate the replica data of the migrated data, the replica data can be accurately routed to the target extended data shard, ensuring the accuracy of data migration. In this way, by simplifying user operations, reusing the sharding deployment configuration information of the source data shards to accelerate the creation of extended databases, and completing the migration of replica data based on the data sharding field, the automation level of the database expansion process is improved.

[0069] The steps described above are explained in detail below.

[0070] First, regarding step 210, the number of extended data shards in this embodiment refers to the total number of data shards planned to be added during the database expansion process.

[0071] In one example, the number of extended data shards is entered manually by the user. For instance, the user can directly enter the number of extended data shards through the command-line interface provided by the database cloud platform; or, for example, the user can enter the number of extended data shards through the graphical user interface provided by the database cloud platform.

[0072] In another example, the number of extended data shards can be automatically generated by the database cloud platform. Specifically, the database cloud platform can collect data size information and operational access load information of the database in real time, predict the business load trend of the database within a preset time window through a business load prediction model, and automatically calculate and generate the number of extended data shards to perform database expansion in combination with a preset single shard capacity threshold.

[0073] In yet another example, the number of expanded data shards can be automatically determined by the database expansion system. Specifically, the database expansion system can respond to preset timed expansion rules and automatically determine the pre-configured number of expanded data shards.

[0074] Secondly, regarding step 220, the data scale information in this embodiment may include, but is not limited to: the total amount of data currently stored in the database and data growth trend data. Specifically, the data scale information may include the total amount of data in the database, the amount of new data added within a preset time window, and historical data growth trends. The operational access load information may include the number of data access requests, concurrent connections, CPU utilization, and memory usage that the database experiences during the statistical time period. The operational access load information is used to characterize the actual operational pressure and resource consumption of the database.

[0075] In this embodiment, the source data shard refers to a unit in the database that already exists and is used to store data before the database expansion operation is performed. Each source data shard manages a portion of the data set. The shard deployment configuration information may include the deployment method, resource allocation, network topology, and routing configuration information of the source data shards in the database, as well as related components such as proxy components and management components. The extended database refers to a newly created database instance during the database expansion process, used to carry all the migrated data.

[0076] In this embodiment of the application, in order to accurately allocate appropriate resources to the extended data shards in the extended database so as to ensure that the performance and resource utilization of the expanded database reach the optimal level, step 220 may specifically include steps 2201 and 2202, as shown below.

[0077] Step 2201: Based on the data scale information and operational access load information of the database, determine the sharding resource specification information of the extended data shards in the extended database. The sharding resource specification information includes storage resources, computing resources and network resources.

[0078] Storage resources refer to the amount of storage space and storage medium type allocated to each extended data shard. Computing resources refer to the number of CPU cores, memory size, and other computing power allocated to each extended data shard. Network resources refer to the network bandwidth and number of network connections allocated to each extended data shard.

[0079] In this step, the database's data size and operational load information can be input into a pre-built computing power assessment model. The model processes this information to obtain the resource specifications for the expanded database. Then, based on the number of expanded data shards, the storage, computing, and network resources required for each shard are calculated. Specifically, the computing power assessment model determines the expanded database's resource specifications using the following formulas: Based on the data size and expected growth rate, the required storage capacity is determined; based on the operational load information, the required number of CPU cores and memory size are determined; and based on the concurrent connections and data transfer volume in the operational load information, the required network bandwidth is determined.

[0080] Step 2202: Create an extended database based on the sharding resource specification information and the sharding deployment configuration information of the source data shards in the database.

[0081] Among them, the sharded deployment configuration information is a collection of information on the sharded deployment and operation of source data.

[0082] In this step, the sharding deployment configuration information of the source data shard is used to provide a deployment template for the source data shard in the database, ensuring that the newly created extended data shard is consistent with the existing source data shard in terms of hardware configuration, software environment, network topology, etc., thereby ensuring the overall stability of the created extended database.

[0083] Therefore, when creating an extended database, the resource specifications required for the extended data shards can be accurately determined based on the actual data size and operational access load of the database, avoiding resource waste and performance bottlenecks caused by fixed resource configurations. Furthermore, creating the extended database by combining the sharding deployment configuration information of the source data shards ensures consistency in deployment and management between the source and extended data shards, improving the efficiency of database expansion operations.

[0084] In this embodiment, to ensure that the newly created extended data shards can be integrated into the existing database cloud platform and effectively carry out data access requests and daily operation and maintenance management, the shard deployment configuration information includes shard metadata, proxy component configuration information, and switch management component configuration information. Shard metadata is the information required for deploying and scheduling source data shards and performing data partitioning. Shard metadata refers to data describing the logical and physical attributes of database shards, and may include, but is not limited to: the unique identifier of the source data shard, the logical location of the source data shard in the source data shard cluster, data distribution rules, and the table structure carried by the source data shard. Proxy component configuration information may include, but is not limited to: the proxy component's network address, authentication key, shard routing protocol, load balancing strategy, connection pool parameters, and read / write distribution strategy. Switch management component configuration information may include, but is not limited to: the switch management component's network address, monitoring and reporting protocol, heartbeat reporting cycle, status collection indicator list, remote operation and maintenance authentication credentials, and fault detection mechanism.

[0085] Based on this, step 2202 above may specifically include steps 22021 and 22022, as shown below.

[0086] Step 22021: Based on the sharding resource specification information and the sharding metadata of the source data shards in the database, create an initial extended database. The number of initial extended data shards in the initial extended database is equal to the number of extended data shards.

[0087] In this step, the sharding instance creation interface of the database cloud platform can be called to instantiate a new initial extended data shard equal to the number of extended data shards on the database cloud platform, based on the pre-determined sharding resource specification information and the sharding metadata of the source data shards.

[0088] Step 22022: Based on the proxy component configuration information, configure the data routing link between the initial extended data shard in the initial extended database and the proxy component; and based on the switching management component configuration information, configure the management link between the initial extended data shard in the initial extended database and the switching management component, thereby obtaining the extended database.

[0089] The data routing link is used to distribute data access requests for the extended database to the corresponding extended data shards via the proxy component; the management link is used to enable the switching management component to monitor the running status of each extended data shard in the extended database.

[0090] In this step, for the configuration of the data routing link, the network address of the proxy component is included in the access whitelist of the initial extended data shard. Combined with the authentication key, shard routing protocol, load balancing strategy, connection pool parameters, and read / write distribution strategy, a dedicated access account and full read / write permissions for the initial extended database are configured for the proxy component. This completes the identity verification and access authentication of the proxy component, supporting the proxy component in forwarding data access requests for the initial extended data shard based on the shard routing protocol, load balancing strategy, controllable connection pool scheduling, and differentiated read / write distribution rules. This completes the configuration of the data routing link. For the configuration of the management link, a lightweight monitoring agent is deployed within each initial extended data shard. Each initial extended data shard uses the built-in monitoring agent to load and parse the network address of the switching management component, monitoring reporting protocol, heartbeat reporting cycle, status collection indicator list, remote operation and maintenance authentication credentials, and fault detection mechanism, thereby completing the management link configuration.

[0091] Therefore, by reusing shard metadata to create the initial extended database, configuring the corresponding data routing links using proxy component configuration information, and configuring the corresponding management links using switching management component configuration information, the efficiency of extended database creation is improved. Furthermore, the establishment of data routing links ensures that data access requests initiated by clients are accurately distributed to the corresponding extended data shards by the proxy component, guaranteeing the correctness and efficiency of data access. The establishment of management links enables the switching management component to monitor the operational status of the extended data shards in real time.

[0092] Furthermore, regarding step 230, the stored data in this embodiment refers to the actual business data stored in the database. During database expansion, the stored data needs to be reorganized and migrated. The data shard field is a logical identifier used to uniquely identify data and determine which extended data shard the data should be routed to. The data shard field can be the primary key identifier of the stored data. The replica data refers to a complete copy of the data stored in the source data shard. During data migration, the replica data stored in the source data shard is first migrated to the extended database to ensure the data availability of the source database. In this embodiment, the target extended data shard refers to the extended data shard determined according to the data shard field during the data migration process, which is used to receive specific replica data.

[0093] In some embodiments of this application, the number of source data shards is N, where N can be an integer greater than or equal to 1. Based on this, before performing step 230 above, the database expansion method may also include step 240.

[0094] Step 240: Using a data export engine that matches the data type of the database, perform a data export operation on the storage data corresponding to each of the N source data shards to obtain the copy data corresponding to each source data shard, and use the copy data corresponding to each of the N source data shards as the copy data of the storage data.

[0095] The data export engine is a tool specifically designed to extract data from a database and convert it into a specific format. For example, this can be achieved using a database backup tool. The copy data corresponding to the source data shard refers to a copy of the data extracted from the source data shard through a data export operation, which is completely identical to the original data content. These copies can be stored as files in a file system or a dedicated data transfer buffer.

[0096] In this embodiment, selecting a data export engine that matches the data type of the database is to ensure that the data export engine can correctly parse and process the data structure, data format, and storage mechanism specific to the target database. For example, for relational databases, an engine that supports SQL syntax and table structure export can be used.

[0097] Therefore, before migrating the replica data of the stored data to the expanded database, a data export engine matching the database data type is used to perform data export operations on each source data shard to obtain replica data. This ensures that at the start of the data migration, all data to be migrated is a complete and consistent snapshot, avoiding data inconsistencies caused by continuous writes to the source database during the migration process. Furthermore, by decoupling the data export operation from the data migration, the impact of the data migration operation on the online service performance of the source database is reduced, improving business continuity during the database expansion process.

[0098] In some embodiments of this application, in order to ensure that the migration data can still be correctly routed and accessed in the expanded database after expansion, the above step 230 may specifically include steps 2301 and 2302.

[0099] Step 2301: Based on the data sharding field of the stored data corresponding to the source data shard and the extended data sharding routing rules of the extended database, determine the target extended data shard corresponding to the replica data of the stored data from the extended database. The extended data sharding routing rules are determined based on the source data sharding routing rules of the database.

[0100] Among them, the extended data sharding routing rule refers to the rules for redistributing data in the extended database. The source data sharding routing rule refers to the rules for distributing data in the source database.

[0101] In this embodiment, the source data sharding routing rules of the database and the extended data sharding routing rules of the extended database are logically related. The extended data sharding routing rules of the extended database can be adjusted and extended based on the data sharding routing rules of the source database. For example, the source database uses a hash sharding rule of primary key ID%N, where N is the number of source data shards, while the extended database can use a hash sharding rule of primary key ID%M, where M is the number of extended data shards, where M>N.

[0102] In this step, the primary key ID of the replica data is obtained, and the target extended data shard ID corresponding to the data sharding field is calculated using the primary key ID%M.

[0103] Step 2302: Migrate the replica data to the target extended data shard.

[0104] In this step, data import tools such as the MySQL import tool can be used to import the replica data corresponding to each source data shard in the database into the corresponding target extended data shard.

[0105] Therefore, by determining the extended data sharding routing rules based on the source data sharding routing rules, accurate mapping of replica data can be achieved, effectively avoiding data misalignment or routing errors that may occur during data migration due to the extended data sharding routing rules, thereby improving the accuracy and reliability of data migration.

[0106] In some embodiments of this application, during the process of exporting stored data from the source data shards using a data export engine to obtain duplicate data and migrating it to the extended database, the database still needs to provide services to the client. New data writes and changes will continue to occur. If only the full duplicate data is migrated, it may lead to data inconsistency between the extended database and the source database, thereby affecting the data integrity of the extended database. Based on this, after performing step 230 above, the database expansion method may further include steps 2501 to 2503.

[0107] Step 2501: For each of the N source data shards, after completing the storage data export operation of the source data shard, the log position corresponding to the completion time of the storage data export operation is taken as the starting position for incremental data extraction.

[0108] In this step, after completing the data export operation for each source data shard, the log sequence number or timestamp corresponding to the export completion time is queried and used as the incremental data extraction point.

[0109] Step 2502: If there are new logs in the logs of each source data shard after the starting point of incremental data extraction, extract the shard data incremental log entries from the new logs corresponding to each source data shard. The shard data incremental log entries include incremental data and the data shard field of incremental data.

[0110] Incremental data log entries refer to log records in the source data shard log that include data changes. These entries capture all data change operations that occur after the full data export, such as inserts, updates, and deletions. Incremental data represents the actual changed data content, while the data shard field indicates which target data shard the incremental data should be routed to; this can be a primary key ID.

[0111] In this step, the incremental data extraction point serves as the logical starting point for incremental data capture, ensuring a seamless connection between the full data and incremental data and preventing data omissions. Specifically, when capturing incremental data, the binary log file of the database can be parsed using the incremental parsing component to identify data change events and extract the incremental data corresponding to each data change event and the corresponding data sharding field as sharded data incremental log entries.

[0112] Step 2503: Based on the data sharding field of the incremental data corresponding to each source data shard, migrate the incremental data corresponding to each source data shard to the extended database.

[0113] In this step, incremental log entries can be stored in an incremental queue. Then, based on the data import tool provided by the database that corresponds to the data type of the extended database, the incremental data in the incremental queue can be imported into the corresponding extended data shard in the extended database according to the extended data sharding routing rules.

[0114] The following specific example will be used to illustrate steps 2501 to 2503 above.

[0115] During database expansion, if the source database uses MySQL and is configured with binary logs (binlog), after the mysqldump tool completes the data export operation for each of the N source data shards, the database expansion system immediately reads the current binlog filename and binlog position of that source data shard and records this as the starting point for incremental data extraction. A separate incremental data capture service continuously monitors binlog updates for that source data shard. Once a new write occurs to the binlog file and its position exceeds the binlog position, the incremental data capture service calls the incremental parsing component to parse these new binlog events. For each parsed event, the incremental parsing component extracts the actual changed data content, i.e., the incremental data. Based on the primary key identifier of this incremental data and the extended data shard routing rules, it calculates the shard routing identifier of the target extended data shard to which the incremental data should be routed. Then, based on the shard routing identifier of the incremental data, it imports the incremental data into the corresponding extended data shard in the extended database.

[0116] Therefore, after the copy data of the source data shards is exported, the log position is recorded as the starting point for incremental data extraction, and the incremental data generated thereafter is continuously captured and migrated. This ensures that the extended database can synchronize the latest state of the source database in real time and accurately. Without interrupting external services, it avoids data loss or inconsistency caused by data changes during data migration, and greatly improves data integrity during the expansion process.

[0117] In some embodiments of this application, if the migration of incremental data and the migration of replica data do not have a clear order, it may lead to data inconsistency during the data migration process. Based on this, step 2503 above may specifically include: after completing the data migration operation of the replica data corresponding to each source data shard, migrating the incremental data corresponding to each source data shard to the extended database according to the data shard field of the incremental data corresponding to each source data shard.

[0118] The following is a detailed explanation of step 2503 using a specific example.

[0119] If a database system comprises two source data shards, source data shard 0 and source data shard 1, and four extended data shards, extended data shard 0, extended data shard 1, extended data shard 2, and extended data shard 3, during data migration, once all replica data of source data shard 0 has been successfully migrated to extended data shard 0 and extended data shard 2, the database expansion system will update the migration status of source data shard 0 to "replica data migration complete." At this point, all incremental data generated after the replica data export from source data shard 0 will be migrated to either extended data shard 0 or extended data shard 2 in the extended database, based on the corresponding data shard field. Incremental data migration for source data shard 1 will only begin after the replica data migration of source data shard 1 is complete.

[0120] Therefore, by limiting incremental data migration to only after the corresponding source data shard's replica data has been completely migrated to the extended database, the potential data inconsistency between replica data migration and incremental data migration during database expansion can be effectively resolved, thereby significantly improving data integrity and consistency during database expansion.

[0121] In some embodiments of this application, during the data migration process, in order to ensure that the data access requests initiated by the client can be processed correctly and without interruption, and to smoothly switch service traffic from the source database to the extended database, the database expansion method may further include steps 2601 and 2602.

[0122] Step 2601: During the replica data migration process, in response to the data access request initiated by the client, the proxy component routes the data access request to the source data shard of the database according to the source data sharding routing rules of the database.

[0123] In this context, a client-initiated data access request refers to a request from an external client to perform read and write operations on the database, which may include, but is not limited to, data query operations, data insertion operations, data update operations, and data deletion operations. The database source data shard routing rule refers to determining the source data shard to which the data belongs based on a hash calculation using the primary key identifier. For example, if the database has two source data shards, source data shard 0 and source data shard 1, the shard routing rule may include: performing a modulo operation of ID%2 on the primary key ID, and determining the source data shard to which the data belongs based on the modulo operation result. For example, when the modulo operation result is 0, the data belongs to source data shard 0; when the modulo operation result is 1, the data belongs to source data shard 1.

[0124] In this step, the data migration process refers to the entire process of transferring the stored data replica to the extended database from the beginning until the transfer is complete. During the data migration, the proxy component parses the data access requests initiated by the client according to the sharding routing rules used by the database and forwards them to the source data shard storing the corresponding data for processing.

[0125] Step 2602: After completing the incremental data migration of the source data shards in the database, switch the effective routing rule of the proxy component from the source data shard routing rule of the database to the extended data shard routing rule of the extended database.

[0126] In this step, after completing the incremental data migration of the source data shards in the database, the data state of the extended database remains consistent with that of the source database and reaches a switchable state. At this point, the proxy component can update the internally maintained routing configuration, modifying the original source data shard routing rules that pointed to the source data shards to extended data shard routing rules that point to the extended data shards in the extended database. For example, if the extended database has four extended data shards: extended data shard 0, extended data shard 1, extended data shard 2, and extended data shard 3, the shard routing rules could include: performing a modulo operation of ID%4 on the primary key ID, and determining the extended data shard to which the data belongs based on the modulo operation result. For example, when the modulo operation result is 0, the data belongs to extended data shard 0; when the modulo operation result is 1, the data belongs to extended data shard 1; when the modulo operation result is 2, the data belongs to extended data shard 2; and when the modulo operation result is 3, the data belongs to extended data shard 3.

[0127] Therefore, during data migration, client-initiated data access requests can be continuously processed, avoiding service downtime or interruption and greatly improving user experience and database system availability. Furthermore, the intelligent routing switching mechanism of the proxy component enables a smooth transition from the source database to the extended database, ensuring the continuity and consistency of data access.

[0128] In some embodiments of this application, during the database expansion process, if the newly created extended database has configuration differences from the original database environment, the expanded extended database may be unable to provide services normally, or even cause business interruption. Based on this, the database expansion method may further include steps 2701 and 2702.

[0129] Step 2701: Before performing the replica data migration operation of the stored data, perform consistency verification on the sharding auxiliary configuration information of the extended data shard and the sharding auxiliary configuration information of the source data shard to obtain the consistency verification result. The consistency verification result indicates that the consistency verification is successful, triggering the execution of migrating the replica data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shard. The sharding auxiliary configuration information includes information used to support the corresponding database to provide data read and write services.

[0130] The sharding-related configuration information refers to the configuration parameters and metadata used to ensure that the extended data shards can provide normal data read and write services. This configuration information may include, but is not limited to: database configuration parameters, table structure information, user permission information, and resource configuration types. For example, database configuration parameters may include, but are not limited to: database character set, database instance service listening port, account permission set, and table storage engine. Table structure information may include, but is not limited to: table name, field type, indexes, and constraints; user permission information may include, but is not limited to: read and write permissions, and administrative permissions; resource configuration types may include, but are not limited to: storage type and compute instance specifications.

[0131] Step 2702: After the data migration operation of the database is completed, perform a data migration integrity check on the extended database to obtain an integrity check result. The integrity check result indicates that the integrity check has passed, triggering the execution of routing data access requests for the source data shard to the extended data shard in the extended database.

[0132] In steps 2701 and 2702, after creating the extended database based on the database's data size information, runtime access load information, and the shard deployment configuration information of the source data shard, and before executing the data migration operation for the stored data replicas, a consistency comparison is performed between the shard attachment configuration information of the extended data shard and the shard attachment configuration information of the source data shard. This ensures the environmental consistency between the newly created extended data shard and the source data shard, thereby avoiding data migration failure due to environmental differences. Only when the consistency check passes, confirming that the extended database has the basic conditions to carry the replica data of the source data shard, will the data migration process for the stored data replicas begin. Next, after all the data migration operations for the stored data replicas are completed, to further ensure the expansion quality and data accuracy of the extended database, a data migration integrity check can be performed on the extended database to verify whether the data in the extended database is consistent with the data in the database. Only when the data migration integrity check result indicates that all data in the database has been accurately migrated to the extended database will the operation of routing data access requests for the source data shard to the extended data shard in the extended database be triggered.

[0133] Therefore, by performing consistency checks on sharded auxiliary configuration information before data migration during database expansion, data migration failures or post-expansion service anomalies caused by mismatched configurations between the expanded and source database environments can be effectively avoided, significantly reducing the risk of database expansion failure. Furthermore, performing data migration integrity checks after data migration ensures that all database-related data is accurately and completely migrated to the expanded database, thus avoiding data loss or inconsistency. This dual-check mechanism effectively mitigates potential configuration deviations, data loss, or data inconsistencies during database expansion.

[0134] In some embodiments of this application, in order to comprehensively and accurately perform consistency verification of sharded auxiliary configuration information, the consistency verification result includes at least one of the following: a first verification result, a second verification result, a third verification result, and a fourth verification result. The first verification result is obtained by comparing database configuration parameters, indicating whether the database configuration parameters are consistent. The second verification result is obtained by comparing data table structure information, indicating whether the data table structure information is consistent. The third verification result is obtained by comparing user permission information, indicating whether the user permission information is consistent. The fourth verification result is obtained by comparing resource configuration types, indicating whether the resource configuration types are consistent. Based on this, the above step 2701 may specifically include steps 27011 to 27014.

[0135] Step 27011: Compare the database configuration parameters of the extended data shard with those of the source data shard to obtain the first verification result.

[0136] In this step, the database character set, database instance service listening port, account permission set, and data table storage engine of the extended data shard are compared item by item to see if they are consistent with those of the source data shard. If all configuration items are consistent, the first verification result is determined to be the result indicating that the database configuration parameters are consistent.

[0137] Step 27012: Compare the data table structure information of the extended data shard with the data table structure information of the source data shard to obtain the second verification result.

[0138] In this step, the table name, field type, index, and constraints of the extended data shard are compared with those of the source data shard to see if they are consistent. If all configuration items are consistent, the second verification result is determined to be the result indicating that the data table structure information is consistent.

[0139] Step 27013: Compare the user permission information of the extended data fragment with the user permission information of the source data fragment to obtain the third verification result.

[0140] In this step, the read / write permissions and management permissions of the extended data shard are compared item by item to see if they are consistent with those of the source data shard. If all permission comparisons are consistent, the third verification result is determined to be the result indicating that the user permission information is consistent.

[0141] Step 27014: Compare whether the resource configuration type of the extended data fragment is consistent with the resource configuration type of the source data fragment to obtain the fourth verification result.

[0142] In this step, the storage type and compute instance specifications of the extended data shard are compared with those of the source data shard to see if they are consistent. If all resource configuration types are consistent, the fourth verification result is determined to be the result indicating that the resource configuration types are consistent.

[0143] Therefore, by comparing the database configuration parameters, table structure information, user permission information, and resource configuration types of the extended data shards and the source data shards one by one, and performing consistency checks based on multi-dimensional information during pre-migration, the security of the database expansion process is greatly enhanced. Furthermore, it can effectively identify and correct potential configuration differences between the extended and source databases, avoiding data garbled characters due to character set mismatches, data write failures due to inconsistent table structures, service interruptions due to missing user permissions, and performance bottlenecks due to different resource configurations. This effectively ensures that the extended database can provide the same functionality as the source database when taking over business operations, significantly reducing the risks of database expansion operations.

[0144] In some embodiments of this application, the integrity verification result includes at least one of the following: a fifth verification result, a sixth verification result, a seventh verification result, and an eighth verification result; the fifth verification result is obtained by comparing the configuration parameters of the migrated database, indicating whether the configuration parameters of the migrated database are consistent with the configuration parameters of the source data shard database. The sixth verification result is obtained by comparing the total amount of data, indicating whether the total amount of data verification has passed. The seventh verification result is obtained by verifying the extended data shard routing rules, indicating whether the shard routing rules are correctly effective. The eighth verification result is obtained by comparing the data content of the sample data, indicating whether the data content verification has passed. Based on this, step 2702 may specifically include steps 27021 to 27024.

[0145] Step 27021: Compare the database configuration parameters of the migrated extended data shards in the extended database with the database configuration parameters of the source data shards to obtain the fifth verification result.

[0146] In this step, the character set of the migrated database after the extended data sharding, the listening port of the migrated database instance service, the set of the migrated account permissions, and the storage engine of the migrated data table are compared item by item to see if they are consistent with those of the source data sharding. If all configuration items are consistent, the fifth verification result is determined to be the result indicating that the database configuration parameters are consistent.

[0147] Step 27022: Compare the total amount of data after migration in the extended database with the total amount of data corresponding to the migrated data in the database to obtain the sixth verification result.

[0148] Step 27023: According to the extended data sharding routing rules of the extended database, verify whether the first sample data belongs to the corresponding reference extended data shard, and obtain the seventh verification result.

[0149] In this step, a batch of first sample data is randomly selected from the source database. Then, the reference extended data shards where these sample data should be stored are calculated according to the extended data sharding routing rules of the extended database. The reference extended data shards of the extended database are actually queried to see if there are indeed first sample data in the reference extended data shards of the extended database. If there are first sample data in the reference extended data shards of the extended database, the seventh verification result is determined to be the result that the sharding routing rules are correctly effective.

[0150] Step 27024: Compare the data content of the second sample data in the extended database with that of the third sample data in the database to see if they are consistent, and obtain the eighth verification result: the data sharding fields of the second sample data and the third sample data are the same.

[0151] In this step, corresponding data records can be extracted from the source database and the extended database based on the same data sharding field, i.e., primary key ID, as the second and third sample data. Then, the data content is compared field by field and row by row. If the data content of the second and third sample data is consistent, the eighth verification result is determined to be the result that the data content verification has passed.

[0152] Therefore, after the database migration operation is completed, the consistency of configuration parameters between the expanded database and the source database is ensured through post-migration database configuration parameter verification, avoiding service anomalies caused by environmental differences; the integrity of the data migration is effectively verified through total data comparison, reducing the risk of data loss; sample data routing verification ensures that the new sharding routing rules are correctly applied, guaranteeing the accuracy of data access request routing; and sample data content comparison further verifies the accuracy of the migrated data, preventing data tampering or corruption during the migration process. In this way, through multi-dimensional data verification, the data consistency between the expanded database and the source database can be effectively guaranteed, thereby ensuring a smooth transition during the database expansion process and the stable operation of the expanded database.

[0153] In some embodiments of this application, after the data migration operation is completed and the integrity verification passes, data access requests for the source data shard are routed to the extended data shard in the extended database. However, directly switching all data access requests to the new extended database at once may pose potential risks, thereby affecting service continuity. Based on this, in step 2702 above, where the integrity verification result indicates that the integrity verification has passed, the database expansion method may further include steps 27024 and 27025, as detailed below.

[0154] Step 27024: If the integrity verification result indicates that the integrity verification has passed, the data access requests for the source data shards at a preset ratio will be routed to the extended data shards in the extended database, and the service operation indicators of the extended data shards within the preset time window will be detected.

[0155] Service performance metrics may include, but are not limited to: CPU utilization, memory utilization, queries per second, transaction processing time, average query latency, error rate, and number of connections. The preset time window can be a few minutes, hours, or days; the specific value can be set based on business sensitivity and user observation needs, and is not specifically limited here. The preset percentage can be 50%.

[0156] In this step, routing rules can be configured through the proxy component to route a preset proportion of data access requests to the extended database based on any of the following data access request extraction methods, in order to verify the performance of the extended database under actual load: the source of the data access request, user identifier, request type, and random sampling. Next, real-time monitoring of the extended database is collected using the database's built-in monitoring tools or third-party application performance management tools to obtain service operation metrics of the extended database within a preset time window, in order to evaluate whether the extended database can stably and efficiently provide services to clients.

[0157] Step 27025: If the value of the service operation metric is within the preset metric value compliance range, the full data access request for the source data shard will be routed to the extended data shard in the extended database.

[0158] The preset compliance range for indicator values ​​refers to the threshold range set for various service operation indicators. For example, average query latency is less than 20ms, and error rate is less than 0.1%.

[0159] In this step, when all service operation metrics are within the corresponding preset metric value compliance range, the routing rules of the proxy component are updated to route all data access requests pointing to the source data shard to the extended database. If any service operation metric is not within the corresponding preset metric value compliance range, the switching of all data access requests is paused, and the process returns to step 2702.

[0160] Therefore, during database expansion, by first routing a preset proportion of data access requests to the extended database and monitoring service performance metrics in real time, the actual operational status of the extended database can be verified under limited business traffic. Only when all service performance metrics are within the preset compliance range will a full-scale business traffic switch be implemented. This significantly improves the stability and reliability of the extended database expansion process, effectively reducing the risk of service interruptions or performance degradation caused by database expansion operations.

[0161] In some embodiments of this application, if the source data shard remains active after performing step 27025, it may lead to resource waste and potential data inconsistency risks. Therefore, the database expansion method may further include steps 2801 to 2804, as detailed below.

[0162] Step 2801: Perform an offline isolation operation on the source data shards in the database. The offline isolation operation includes at least one of the following: deleting the shard routing rule corresponding to the source data shard in the proxy component that communicates with the database; disconnecting the data transmission session for the source data shard; and closing the data access port of the source data shard.

[0163] In this step, the routing rules related to the source data shard in the proxy component can be deleted by calling the routing management interface of the proxy component, so as to ensure that all new data access requests are no longer routed to the source data shard that has completed data migration.

[0164] Disconnecting a data transmission session for a source data shard can be achieved in any of the following ways: terminating all current network connections and data interactions with the source data shard via connection management tools, sending a session termination signal, or forcibly closing the network connection.

[0165] Disabling the data access port for source data shards can be achieved by adjusting the database service listening port settings or configuring an access control list on the network device to block any new connection attempts at the network level.

[0166] Step 2802: Revoke the authorized account's operation permissions for the source data shards and switch the working mode of the source data shards from read-write mode to forced read-only mode.

[0167] In this step, revoking an authorized account's access permissions for the source data shards can be achieved by modifying the database's user permission configuration. Switching the operating mode of the source data shards can be done by executing database configuration commands or modifying the database instance's configuration file.

[0168] Step 2803: If the source data shard meets the preset safe shutdown conditions, terminate the instance process of the source data shard. The preset safe shutdown conditions include at least one of the following: no background tasks are running, no data write operations are running, and there is no log data to be flushed in the log buffer.

[0169] In this step, after confirming by querying the internal state of the database that there are no background tasks being executed, no data write operations in progress, and no log data that has not yet been persisted in the log buffer, the instance process of the source data shard is terminated through the instance termination application programming interface provided by the container orchestration tool or the cloud platform.

[0170] Step 2804: Perform resource release operations on the source data fragments.

[0171] Among them, resources refer to cloud platform resources, which may include, but are not limited to: computing resources such as CPU, memory, cloud hosts, storage resources such as data disks, backup storage, and network resources such as virtual IPs, network ports, and network bandwidth.

[0172] In this step, resource release operations can be performed on the source data shards through the resource management API provided by the cloud platform, specifically involving the reclamation of all computing resources, storage resources, and network resources occupied by the source data shards.

[0173] Therefore, after the database is expanded and traffic switching is completed, the unused source data shards are sequentially subjected to the steps of offline isolation, permission revocation, instance process shutdown, and resource reclamation. This not only avoids resource idleness and waste, but also effectively prevents data inconsistency through offline isolation and instance process shutdown operations, ensuring data consistency throughout the entire database expansion process.

[0174] Based on the same inventive concept Figure 3 This is a schematic diagram of the structure of a database expansion device provided in one embodiment of this application.

[0175] In some embodiments of this application, Figure 3 The database expansion device shown can be installed on a computer device.

[0176] like Figure 3 As shown, the database expansion device 30 may specifically include: The acquisition module 301 is used to acquire the number of extended data shards used to perform database expansion; The creation module 302 is used to create an extended database based on the data scale information, running access load information and the sharding deployment configuration information of the source data shards in the database. The number of extended data shards included in the extended database is the number of extended data shards. The migration module 303 is used to migrate the copy data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shard.

[0177] In this way, users do not need to manually generate complex migration task lists; they only need to obtain the number of extended data shards through module 301 to start the database expansion process. When creating the extended database, module 302 reuses the shard deployment configuration information of the source data shards, improving the creation efficiency of the extended database. Simultaneously, it comprehensively considers the database's data size and operational access load information, achieving dynamic optimization of database resources and effectively avoiding resource idleness and waste. Subsequently, the migration module 303 uses a method based on the data shard field of the migration data itself to migrate the replica data, ensuring that the replica data is accurately routed to the target extended data shard, guaranteeing the accuracy of data migration. Thus, by simplifying user operations, reusing the shard deployment configuration information of the source data shards to accelerate the creation of the extended database, and completing the replica data migration based on the data shard field, the automation level of the database expansion process is improved.

[0178] The database expansion device 30 in the embodiments of this application will be described in detail below.

[0179] In one or more optional embodiments, the creation module 302 is specifically used to determine the sharding resource specification information of the extended data shards in the extended database based on the data scale information and running access load information of the database. The sharding resource specification information includes storage resources, computing resources and network resources. An extended database is created based on the sharded resource specification information and the sharded deployment configuration information of the source data shards in the database.

[0180] In one or more optional embodiments, the creation module 302 is specifically used to, when the shard deployment configuration information includes shard metadata, proxy component configuration information and switching management component configuration information, and the shard metadata is information required for deploying, scheduling the source data shards and performing data partitioning, create an initial extended database based on the shard resource specification information and the shard metadata of the source data shards in the database, wherein the number of initial extended data shards in the initial extended database is the number of extended data shards; Based on the proxy component configuration information, configure a data routing link between the initial extended data shard in the initial extended database and the proxy component; and based on the switching management component configuration information, configure a management link between the initial extended data shard in the initial extended database and the switching management component, thereby obtaining the extended database; The data routing link is used to distribute data access requests for the extended database to the corresponding extended data shards via the proxy component; the management link is used to enable the switching management component to monitor the running status of each extended data shard in the extended database.

[0181] In one or more optional embodiments, the database expansion device 30 may further include a data export module, which, when the number of source data shards is N, and before performing the step of migrating the copy data of the stored data to the expanded database according to the data shard field of the stored data corresponding to the source data shards, performs a data export operation on the stored data corresponding to each of the N source data shards through a data export engine that matches the data type of the database, to obtain the copy data corresponding to each source data shard, and uses the copy data corresponding to each of the N source data shards as the copy data of the stored data.

[0182] In one or more optional embodiments, the database expansion device 30 may further include a determining module, which is used to, for each of the N source data shards, take the log position corresponding to the completion time of the storage data export operation as the starting position for incremental data extraction when the storage data export operation of the source data shard is completed. The extraction module is used to extract incremental log entries for shard data from the new logs corresponding to each source data shard when there are new logs in the logs of each source data shard after the incremental data extraction start point. The incremental log entries for shard data include incremental data and the data shard field of the incremental data. The migration module is also used to migrate the incremental data corresponding to each source data shard to the extended database according to the data shard field of the incremental data corresponding to each source data shard.

[0183] In one or more optional embodiments, the determining module may be specifically used to migrate the incremental data corresponding to each source data shard to the extended database according to the data shard field of the incremental data corresponding to each source data shard after the data migration operation of the replica data corresponding to each source data shard is completed.

[0184] In one or more optional embodiments, the database expansion device 30 may further include a routing module, which, in response to a data access request initiated by a client during the replica data migration process, routes the data access request to the source data shard of the database through a proxy component according to the source data shard routing rules of the database. The switching module is used to switch the effective routing rule of the proxy component from the source data shard routing rule of the database to the extended data shard routing rule of the extended database after the incremental data migration of the source data shards in the database is completed. In one or more optional embodiments, the migration module 303 may be specifically used to determine the target extended data shard corresponding to the replica data of the stored data from the extended database according to the data shard field of the stored data corresponding to the source data shard and the extended data shard routing rules of the extended database, wherein the extended data shard routing rules are determined based on the source data shard routing rules of the database. The replica data is migrated to the target extended data shard.

[0185] In one or more optional embodiments, the database expansion device 30 may further include a verification module, used to perform consistency verification on the sharding auxiliary configuration information of the extended data shard and the sharding auxiliary configuration information of the source data shard before performing the replica data migration operation of the stored data, and obtain a consistency verification result, wherein the consistency verification result indicates that the consistency verification is passed, triggering the execution of migrating the replica data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shard, and the sharding auxiliary configuration information includes information for supporting the corresponding database to provide data read and write services; Upon completion of the data migration operation of the database, a data migration integrity check is performed on the extended database to obtain an integrity check result. The integrity check result indicates that the integrity check has passed, triggering the execution of data access requests for the source data shard to be routed to the extended data shard in the extended database.

[0186] In one or more optional embodiments, the verification module can also be used to compare whether the database configuration parameters of the extended data shard and the database configuration parameters of the source data shard are consistent when the shard-related configuration information includes database configuration parameters, data table structure information, user permission information and resource configuration type; and the consistency verification result includes at least one of the following: a first verification result, a second verification result, a third verification result, and a fourth verification result, to obtain a first verification result. A second verification result is obtained by comparing whether the data table structure information of the extended data shard is consistent with the data table structure information of the source data shard. A third verification result is obtained by comparing whether the user permission information of the extended data fragment is consistent with the user permission information of the source data fragment. The fourth verification result is obtained by comparing whether the resource configuration type of the extended data fragment is consistent with the resource configuration type of the source data fragment.

[0187] In one or more optional embodiments, the verification module can also be used to compare whether the database configuration parameters of the migrated extended data shard in the extended database are consistent with the database configuration parameters of the source data shard when the integrity verification result includes at least one of the following: the fifth verification result, the sixth verification result, the seventh verification result, and the eighth verification result, so as to obtain the fifth verification result; The sixth verification result is obtained by comparing the total amount of data after migration in the extended database with the total amount of data corresponding to the migrated data in the database. According to the extended data sharding routing rules of the extended database, verify whether the first sample data belongs to the corresponding reference extended data shard, and obtain the seventh verification result; The data content of the second sample data in the extended database is compared with the data content of the third sample data in the database to determine whether they are consistent. The eighth verification result is obtained, and the data sharding fields of the second sample data and the third sample data are the same.

[0188] In one or more optional embodiments, the routing module can also be used to, when the integrity verification result indicates that the integrity verification has passed, route data access requests for a preset proportion of the source data shard to the extended data shard in the extended database, and detect the service operation indicators of the extended data shard within a preset time window. If the value of the service operation metric is within the preset metric value compliance range, the full data access request for the source data shard will be routed to the extended data shard in the extended database.

[0189] In one or more optional embodiments, the database expansion device 30 may further include a resource release module, used to perform an offline isolation operation on the source data shard in the database after executing the step of routing the full data access request for the source data shard to the extended data shard in the extended database, wherein the offline isolation operation includes at least one of the following: deleting the source data shard routing rule related to the source data shard in the proxy component; disconnecting the data transmission session for the source data shard; and closing the data access port of the source data shard; Revoke the authorized account's operation permissions for the source data shard, and switch the working mode of the source data shard from read-write mode to forced read-only mode; If the source data shard meets the preset safe shutdown conditions, the instance process of the source data shard is terminated. The preset safe shutdown conditions include at least one of the following: no background tasks are being executed, no data write operations are being executed, and there is no log data to be flushed in the log buffer. Perform resource release operations on the source data fragments.

[0190] Based on the same inventive concept, this application also provides a computer device. (Specifically combined with...) Figure 4 Please provide a detailed explanation.

[0191] Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application.

[0192] like Figure 4 As shown, the computer device 40 may include a processor 401 and a memory 402 storing computer program instructions.

[0193] Specifically, the processor 401 may include a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0194] Specifically, the processor 401 may include a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0195] Memory 402 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory. In a particular embodiment, memory 402 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0196] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the database expansion methods in the above embodiments.

[0197] In one example, computer device 40 may also include a communication interface 403 and a bus 410. Wherein, as... Figure 4 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.

[0198] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0199] Bus 410 includes hardware, software, or both, that couples components of a flow control device together. For example, and not limited to, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count Bus (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0200] The computer device 40 can execute the database expansion method in the embodiments of this application, thereby achieving the combination Figure 2 and Figure 3 The described database expansion method and apparatus.

[0201] Furthermore, in conjunction with the database expansion methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the database expansion methods in the above embodiments.

[0202] Furthermore, in conjunction with the database expansion methods in the above embodiments, this application embodiment can provide a computer program product for implementation. This program product is stored in a storage medium and may specifically include a computer program or instructions. When executed by a processor, the computer program or instructions implement any of the database expansion methods in the above embodiments. This program product is executed by at least one processor to implement the various processes of the database expansion method embodiments described above, and can achieve the same technical effects. To avoid repetition, further details are omitted here.

[0203] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0204] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0205] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0206] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A database expansion method, characterized in that, include: Get the number of extended data shards used to perform database expansion; Based on the data size information, running access load information, and sharding deployment configuration information of the source data shards in the database, an extended database is created, and the number of extended data shards included in the extended database is the number of extended data shards. Based on the data sharding field of the storage data corresponding to the source data shard, the copy data of the storage data is migrated to the extended database.

2. The method according to claim 1, characterized in that, The step of creating an extended database based on the data size information, runtime access load information, and sharding deployment configuration information of the source data shards in the database includes: Based on the data size information and operational access load information of the database, the sharding resource specification information of the extended data shards in the extended database is determined. The sharding resource specification information includes storage resources, computing resources and network resources. An extended database is created based on the sharded resource specification information and the sharded deployment configuration information of the source data shards in the database.

3. The method according to claim 2, characterized in that, The sharding deployment configuration information includes sharding metadata, proxy component configuration information, and switching management component configuration information. The sharding metadata is the information required for deploying and scheduling the source data shards and performing data partitioning. The step of creating an extended database based on the sharded resource specification information and the sharded deployment configuration information of the source data shards in the database includes: Based on the fragmentation resource specification information and the fragmentation metadata of the source data fragments in the database, an initial extended database is created, wherein the number of initial extended data fragments in the initial extended database is the same as the number of extended data fragments. Based on the proxy component configuration information, configure a data routing link between the initial extended data shard in the initial extended database and the proxy component; and based on the switching management component configuration information, configure a management link between the initial extended data shard in the initial extended database and the switching management component, thereby obtaining the extended database; The data routing link is used to distribute data access requests for the extended database to the corresponding extended data shards via the proxy component; the management link is used to enable the switching management component to monitor the running status of each extended data shard in the extended database.

4. The method according to claim 1, characterized in that, The number of source data fragments is N; Before migrating the copy data of the stored data to the extended database based on the data sharding field of the stored data corresponding to the source data shard, the method further includes: Using a data export engine that matches the data type of the database, a data export operation is performed on the storage data corresponding to each of the N source data shards to obtain the replica data corresponding to each source data shard. The replica data corresponding to each of the N source data shards is then used as the replica data of the storage data.

5. The method according to claim 4, characterized in that, The method further includes: For each of the N source data shards, after the storage data export operation of the source data shard is completed, the log position corresponding to the completion time of the storage data export operation is taken as the starting position for incremental data extraction. If there are new logs in the logs of each source data shard after the incremental data extraction start point, extract shard data incremental log entries from the new logs corresponding to each source data shard. The shard data incremental log entries include incremental data and the data shard field of the incremental data. Based on the data sharding field of the incremental data corresponding to each source data shard, the incremental data corresponding to each source data shard is migrated to the extended database.

6. The method according to claim 5, characterized in that, The step of migrating the incremental data corresponding to each source data shard to the extended database based on the data sharding field of the incremental data corresponding to each source data shard includes: After completing the data migration operation of the replica data corresponding to each source data shard, the incremental data corresponding to each source data shard is migrated to the extended database according to the data shard field of the incremental data corresponding to each source data shard.

7. The method according to claim 1, characterized in that, The method further includes: During the replication data migration process, in response to a data access request initiated by the client, the proxy component routes the data access request to the source data shard of the database according to the source data shard routing rules of the database. Upon completion of the incremental data migration of the source data shards in the database, the effective routing rule of the proxy component is switched from the source data shard routing rule of the database to the extended data shard routing rule of the extended database.

8. The method according to claim 1, characterized in that, The step of migrating the replica data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shard includes: Based on the data sharding field of the storage data corresponding to the source data shard and the extended data sharding routing rules of the extended database, the target extended data shard corresponding to the replica data of the storage data is determined from the extended database, and the extended data sharding routing rules are determined based on the source data sharding routing rules of the database; The replica data is migrated to the target extended data shard.

9. The method according to claim 1, characterized in that, The method further includes: Before performing the replica data migration operation of the stored data, a consistency check is performed on the sharding auxiliary configuration information of the extended data shard and the sharding auxiliary configuration information of the source data shard to obtain a consistency check result. The consistency check result indicates that the consistency check has passed, triggering the execution of migrating the replica data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shard. The sharding auxiliary configuration information includes information for supporting the corresponding database to provide data read and write services. Upon completion of the data migration operation of the database, a data migration integrity check is performed on the extended database to obtain an integrity check result. The integrity check result indicates that the integrity check has passed, triggering the execution of data access requests for the source data shard to be routed to the extended data shard in the extended database.

10. The method according to claim 9, characterized in that, The sharding-related configuration information includes database configuration parameters, data table structure information, user permission information, and resource configuration types; The consistency verification result includes at least one of the following: a first verification result, a second verification result, a third verification result, and a fourth verification result; The consistency verification of the fragment attachment configuration information of the extended data fragment and the fragment attachment configuration information of the source data fragment, to obtain the consistency verification result, includes: The database configuration parameters of the extended data shard and the database configuration parameters of the source data shard are compared to determine whether they are consistent, and a first verification result is obtained. A second verification result is obtained by comparing whether the data table structure information of the extended data shard is consistent with the data table structure information of the source data shard. A third verification result is obtained by comparing whether the user permission information of the extended data fragment is consistent with the user permission information of the source data fragment. The fourth verification result is obtained by comparing whether the resource configuration type of the extended data fragment is consistent with the resource configuration type of the source data fragment.

11. The method according to claim 9, characterized in that, The integrity verification result includes at least one of the following: the fifth verification result, the sixth verification result, the seventh verification result, and the eighth verification result; The process of performing data migration integrity verification on the extended database to obtain integrity verification results includes: The fifth verification result is obtained by comparing whether the database configuration parameters of the migrated extended data shard in the extended database are consistent with the database configuration parameters of the source data shard. The sixth verification result is obtained by comparing the total amount of data after migration in the extended database with the total amount of data corresponding to the migrated data in the database. According to the extended data sharding routing rules of the extended database, verify whether the first sample data belongs to the corresponding reference extended data shard, and obtain the seventh verification result; The data content of the second sample data in the extended database is compared with the data content of the third sample data in the database to determine whether they are consistent. The eighth verification result is obtained, and the data sharding fields of the second sample data and the third sample data are the same.

12. The method according to claim 9, characterized in that, The method further includes: If the integrity verification result indicates that the integrity verification has passed, data access requests for a preset proportion of the source data shard will be routed to the extended data shard in the extended database, and the service operation indicators of the extended data shard within the preset time window will be detected. If the value of the service operation metric is within the preset metric value compliance range, the full data access request for the source data shard will be routed to the extended data shard in the extended database.

13. The method according to claim 12, characterized in that, After routing the full data access request for the source data shard to the extended data shard in the extended database, the method further includes: For the source data shards in the database, perform an offline isolation operation, wherein the offline isolation operation includes at least one of the following: deleting the source data shard routing rules related to the source data shard in the proxy component; disconnecting the data transmission session for the source data shard; and closing the data access port of the source data shard. Revoke the authorized account's operation permissions for the source data shard, and switch the working mode of the source data shard from read-write mode to forced read-only mode; If the source data shard meets the preset safe shutdown conditions, the instance process of the source data shard is terminated. The preset safe shutdown conditions include at least one of the following: no background tasks are being executed, no data write operations are being executed, and there is no log data to be flushed in the log buffer. Perform resource release operations on the source data fragments.

14. A database expansion device, characterized in that, include: The acquisition module is used to obtain the number of extended data shards used to perform database expansion; A creation module is used to create an extended database based on the data size information, running access load information, and shard deployment configuration information of the source data shards in the database. The number of extended data shards included in the extended database is the number of extended data shards. The migration module is used to migrate the copy data of the stored data to the extended database according to the data sharding field of the stored data corresponding to the source data shard.

15. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the database expansion method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the database expansion method as described in any one of claims 1-13.

17. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the database expansion method as described in any one of claims 1-13.