Database management method and system, device, and medium

Abstract

Embodiments of the present application are applicable to the technical field of databases, and provide a database management method and system, a device, and a medium. The method comprises: if there is a newly added server, adding configuration information of the newly added server to a global server, the configuration information comprising a newly added server identifier and a first classification identifier, the newly added server identifier being used for uniquely identifying the newly added server, and the first classification identifier being used for identifying data stored in the newly added server; generating a database sharding migration task on the basis of the configuration information, the database sharding migration task carrying a classification identifier of data to be migrated; sending, to a data migration service, the classification identifier of the data to be migrated; and using the data migration service to migrate, to the newly added server, the data to be migrated in the global server. By means of the method, data can be automatically migrated to a newly added server, thereby achieving dynamic expansion of the database management system, and improving the response speed of the database management system.

Description

Database management methods, systems, devices and media

[0001] This application claims priority to Chinese Patent Application No. 202411295892.2, filed on September 14, 2024, entitled "Database Management Method, System, Device and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of database technology, and in particular relates to a database management method, system, device and medium. Background Technology

[0003] When a platform provides services to users, it needs to manage data in order to respond to user requests based on that data. As the number of users using the platform's services increases, the amount of data that the platform needs to manage also increases.

[0004] The sheer volume of data presents challenges to the platform's database management capabilities. As the platform generates large amounts of data daily, the volume of data it needs to retrieve increases, leading to slower response times and a decreased user experience. Summary of the Invention

[0005] In view of this, embodiments of this application provide a database management method, system, device, and medium to achieve dynamic expansion of a data management server cluster and improve the response speed of the data management server cluster.

[0006] The first aspect of this application provides a database management method, including:

[0007] If a new server is added, the configuration information of the new server is added to the global server. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server.

[0008] Based on the configuration information, a database migration task is generated. The database migration task carries a classification identifier of the data to be migrated. The classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated.

[0009] Send the classification identifier of the data to be migrated to the data migration service;

[0010] The data migration service is used to migrate the data to be migrated from the global server to the newly added server.

[0011] A second aspect of this application provides a database management system, including: a server cluster, an application service, a message middleware, a data migration service, and a data push service. The server cluster includes a global server and sub-servers. The global server stores server configuration mapping information, which includes a mapping relationship between the server and the stored data.

[0012] The application service is used to add the configuration information of a newly added server to the global server if a new server is added. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server. Based on the configuration information, a database migration task is generated. The database migration task carries the category identifier of the data to be migrated. The category identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated. The category identifier of the data to be migrated is sent to the data migration service.

[0013] The data migration service is used to migrate the data to be migrated from the global server to the newly added server;

[0014] The application service is used to update the server configuration mapping information in the global server after the database migration task is completed, and to call the data push service to synchronize the updated server configuration mapping information to other application services.

[0015] A third aspect of this application provides a database management apparatus, including:

[0016] An add module is used to add the configuration information of a new server to the global server if a new server is added. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server.

[0017] The generation module is used to generate a database migration task based on the configuration information. The database migration task carries a classification identifier of the data to be migrated. The classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated.

[0018] The sending module is used to send the classification identifier of the data to be migrated to the data migration service;

[0019] The migration module is used to migrate the data to be migrated from the global server to the newly added server through the data migration service.

[0020] A fourth aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0021] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0022] A sixth aspect of this application provides a computer program product that, when run on a computer device, causes the computer device to perform the method described in the first aspect.

[0023] Compared with the prior art, the embodiments of this application have the following advantages:

[0024] The database management method in this application embodiment can be applied to a database management system, which may include a global server. When a new server is added to the database management system, its configuration information can be added to the global server. This configuration information may include the server identifier and category identifier of the new server. The server identifier is used to uniquely identify the new server. The category identifier is used to identify the data stored in the new server. Based on this configuration information, a database sharding migration task can be generated, thereby migrating the corresponding data from the global database to the new server. During the database sharding migration, the category identifier containing the migration data can be sent to a data migration service, which can then migrate the data to be migrated from the global server to the new server. In this application, servers can be dynamically added to the database system, thereby distributing data across various servers. Each server manages the data and responds to user requests, thus improving the response speed of the database management system. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0026] Figure 1 is a schematic diagram of a database management system provided in an embodiment of this application;

[0027] Figure 2 is a flowchart illustrating the steps of a database management method provided in an embodiment of this application.

[0028] Figure 3 is a flowchart illustrating the steps of another database management method provided in an embodiment of this application;

[0029] Figure 4 is a timing diagram of a database management system provided in an embodiment of this application;

[0030] Figure 5 is a flowchart of a database management method provided in an embodiment of this application;

[0031] Figure 6 is a flowchart illustrating the process of adding a server in a database management system according to an embodiment of this application;

[0032] Figure 7 is a schematic diagram of a database management device provided in an embodiment of this application;

[0033] Figure 8 is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0035] The service platform can handle the needs of agents and provide data storage services to multiple connected terminal devices. The platform can store large amounts of data, facilitating service delivery to terminal devices. However, as the number of connected terminal devices increases, the amount of data that needs to be managed also increases, leading to decreased efficiency in responding to terminal device requests.

[0036] For example, as the number of app stores and agents joining the MAXSTORE platform increases, the number of connected mobile devices also surges. When the number of devices on the MAXSTORE platform reaches tens of millions, each device needs to periodically report data to the platform and receive task instructions from the platform. The platform generates hundreds of millions of data points every day, and how to quickly retrieve data and respond to terminal requests while ensuring data isolation and security under the conditions of large data volume and high concurrency has become a significant challenge.

[0037] There are two open-source dynamic database sharding solutions on the market: ShardingJDBC and baomidou-dynamic-datasource. Both of these solutions are based on annotation-based aspects, requiring annotations to be added to all database operations, which is quite intrusive. Furthermore, ShardingJDBC is more complex to configure, and its sharding often involves completely peer-to-peer databases. The MAXSTORE platform library uses a model of one global database + n sub-markets, with each sub-market needing to interact with the global database. Therefore, it involves cross-database operations, meaning the solution needs to support distributed transactions. However, baomidou-dynamic-datasource's support for distributed transactions conflicts with Spring's native transactions, making them incompatible. Meanwhile, MAXSTORE's master-slave database selection is based on Spring's native read-only transactions.

[0038] Based on this, embodiments of this application provide a database management system and a database management method to improve the response efficiency of the service platform and ensure data isolation and security.

[0039] The technical solution of this application will be described below through specific embodiments.

[0040] This embodiment provides a database management system, which may include: a server cluster, application services, message middleware, data migration services, and data push services. The server cluster may include a global server and sub-servers. The global server may store server configuration mapping information, which may include the mapping relationship between servers and stored data. Each server in the server cluster may have a unique server identifier, which can be used to uniquely identify the server. Each category identifier may correspond to a dataset. The server configuration mapping information may be the association between the server identifier and the category identifier of the stored data. It should be noted that each server identifier may be associated with one or more category identifiers.

[0041] The aforementioned application service can be used to add the configuration information of a newly added server to the global server if a new server is added. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server. Based on the configuration information, a database migration task is generated. The database migration task carries the category identifier of the data to be migrated. The category identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated. The category identifier of the data to be migrated is sent to the data migration service.

[0042] The aforementioned data migration service can be used to migrate data from the global server to a newly added server.

[0043] The aforementioned application services can also be used to update server configuration mapping information in the global server after the database migration task is completed, and call the data push service to synchronize the updated server configuration mapping information to other application services.

[0044] Figure 1 is a schematic diagram of a database management system provided in an embodiment of this application. In the database management system shown in Figure 1, the server cluster may include MySQL global and MySQL market. MySQL global is a global database, which can be set up on the aforementioned global server. The global database may include registration information for each terminal device. When a terminal device registers or modifies its registration information, the data needs to be synchronized to the global database.

[0045] The MySQL `market` database is a sub-database, which can reside on a server outside the global server. The sub-database stores data related to terminal devices within a specific market area. Market areas can be divided based on geographical regions or according to a predefined method. Each market area can have a unique market identifier.

[0046] For example, the aforementioned terminal device can be a POS machine. Based on the sales region of the POS machine, it can be divided into domestic market, North American market, European market, etc. The registration information of each POS machine can be stored in the aforementioned global database, and the data of each market can be divided into sub-databases.

[0047] When a database management system (DBMS) first starts providing services, the number of connected terminal devices is generally small, and therefore the amount of data the DBMS needs to manage is also small. At this stage, the DBMS may only include one server. This server can be a global server, upon which the DBMS can receive data from various terminal devices and respond to their requests.

[0048] As the number of connected terminal devices in a database management system increases, the amount of data that needs to be managed also gradually increases. A large amount of data concentrated in a global database reduces data retrieval efficiency. At the same time, data centralization also poses challenges to data security.

[0049] At this point, a server can be added to the database management system, allowing the new server to provide services to a subset of terminal devices.

[0050] When a new server is added to the database management system, its configuration information can be obtained. This configuration information can be set by administrators. The configuration information may include a new server identifier and a first category identifier for the data stored on the new server. The new server identifier uniquely identifies the new server. The first category identifier may correspond to a dataset. For example, the first category identifier can be a market identifier, used to identify the market to which the terminal device is classified. For instance, market identifiers may include 1, 2, 3, ..., n, where 1 can identify the domestic market, 2 the North American market, 3 the European and American markets, etc. Each piece of data stored on the server may include a market identifier to identify the market to which that piece of data belongs.

[0051] Based on the configuration information of the new server, a database sharding migration task can be generated. This task can be used to migrate data from the global database to the new server. The category identifier of the data to be migrated carried by the database sharding migration task can be the first category identifier of the data to be stored on the new server.

[0052] Database management systems can perform data migration through a data migration service. A database sharding migration task can include category identifiers for the data to be migrated. The data migration service can retrieve these category identifiers from the migration task. Based on these identifiers, the service can locate the corresponding first server from the server configuration mapping information in the global database. This first server will be used to store the migrated data after the migration is complete. After determining the first server, the service can create a first database on that server. The service can also search for the migrated data from the global servers based on its category identifiers and then copy the data in batches to the newly created first database on the first server. This first server can be a newly added server.

[0053] It's important to note that when performing a database sharding migration task, the data corresponding to that task can be suspended. That is, during the migration process, the data to be migrated can only be searched; adding, modifying, or deleting data is not allowed. Suspending the data during the migration process helps prevent data corruption.

[0054] Server configuration mapping information can include the mapping relationship between servers and stored data. After data migration is completed, the data storage location changes, and the mapping relationship between servers and stored data will inevitably change as well. Therefore, after the database sharding migration task is completed, the server configuration mapping information in the global server can be updated. Server configuration mapping information can include the mapping relationship between servers and stored data, specifically the association between the server identifier and the category identifier of the stored data. In other words, after data migration is completed based on configuration information, the association between server identifiers and category identifiers can be updated according to the results of the data migration, thereby completing the update of the server configuration mapping information.

[0055] After the server configuration mapping information is updated, the database system can use the data synchronization service to synchronize the server configuration mapping information to all application services in the data system. This is equivalent to instructing each application service to provide services to the terminal devices based on the new server configuration mapping information.

[0056] In one possible implementation, the data in the newly added server may include a category identifier. For example, the configuration information of the newly added server may include server identifier 'a' and category identifier '2', which can be stored in the global server. The data migration service can obtain the category identifier '2' of the database sharding migration task. Based on category identifier '2', the data migration service can determine the target server 'a'. The data migration service can create a new database 'a' on the target server 'a'. The data migration service can search for all data with category identifier '2' in the global server, and then batch copy all data with category identifier '2' to the newly created database 'a', thereby completing the database sharding migration task. After the migration is complete, the association between the global server and category identifier '2' in the server configuration information can be deleted; simultaneously, the association between server identifier 'a' and category identifier '2' can be added to the server configuration mapping information.

[0057] In another possible implementation, the data in the newly added server can include multiple category identifiers. For example, the configuration information of the newly added server can include server identifier 'a' and category identifiers '2' and '3', which can be stored in the global server. The data migration service can obtain category identifiers '2' and '3' for the database sharding migration task. Based on category identifiers '2' and '3', the data migration service can determine the target server 'a'. The data migration service can create databases 'a' and 'b' on the target server 'a'. The data migration service can search for all data with category identifier '2' in the global server and then batch copy all data with category identifier '2' to the newly created database 'a'; similarly, it can search for all data with category identifier '3' in the global server and then batch copy all data with category identifier '3' to the newly created database 'b', thus completing the database sharding migration task. After the migration is complete, the association between the global server and category identifiers '2' and '3' in the server configuration information can be deleted; simultaneously, the association between server identifier 'a' and category identifiers '2' and '3' can be added to the server configuration mapping information.

[0058] Each application service can respond to terminal device requests based on the updated server configuration mapping information. The application service can locate the server containing the data required to respond to the request, and then retrieve the corresponding data from the server's target database, based on the terminal device's identifier and server configuration mapping information.

[0059] In a database management system, servers can be added to a server cluster to automatically migrate data. In this embodiment, servers can be dynamically added to the database system to distribute data across various servers, which then manage the data and respond to user requests, thereby improving the response speed of the database management system.

[0060] Referring to Figure 2, a flowchart illustrating the steps of a database management method provided in this embodiment of the application may specifically include the following steps:

[0061] S201, if a new server is added, the configuration information of the new server is added to the global server. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server.

[0062] In this embodiment, the execution entity can be a computer device, which can be a client device of a server cluster. One or more application services can be deployed on the computer device. Based on these application services, the database management system can send data to various terminal devices and receive service requests from them.

[0063] To reduce the load on the database management system, a server can be added. The newly added server mentioned above is one that is added to the database management system. Initially, this server does not contain any data. Administrators can configure the server to determine which data can be stored and managed.

[0064] The above configuration information can be obtained based on the settings configured by the administrator. For example, the computer device can provide a configuration interface, through which administrators can configure the new server. The computer device can then obtain the administrator's settings data and generate the new server's configuration information. For instance, the computer device can deploy a server management module. When a new server is added, the server management module can display a configuration interface to the user. Administrators can select or fill in the new server's configuration information on the interface. Based on the information entered by the administrator, the computer device can automatically obtain the new server's configuration information. When configuring a new server, the administrator can set a server identifier for it. This server identifier is the aforementioned new server identifier and can be used to uniquely identify the new server. The administrator can also configure a first category identifier for the data stored in the new server. This first category identifier can be used to identify the category corresponding to the data stored in the new server. This category can be determined based on the market to which the terminal device corresponding to the data belongs; that is, the first category identifier can be a market identifier.

[0065] After obtaining the configuration information of the newly added server, the computer device can store the configuration information in the global database, which will facilitate subsequent data migration.

[0066] It should be noted that in this embodiment of the application, each terminal device is registered in the global server, and updates to the registration information of the terminal device also need to be updated in the global server.

[0067] For example, the terminal device can be a POS machine. When the number of POS machines is small, all data can be stored on a global server, which can then centrally manage and handle requests and responses. However, as the sales market expands, the number of terminal devices will increase significantly.

[0068] In different market regions, there may be multiple terminal devices; data from multiple markets is concentrated in one server. When searching for and responding to data, the server will take longer. At this time, the centralized storage of data affects service efficiency.

[0069] Account data from different markets is not highly correlated; for example, POS machine transaction data from North America and Europe generally have little correlation. Therefore, data from different markets can be stored separately and served by different servers. When a new server is added to the server cluster, data from one market can be transferred to the new server, creating a database for that market. When queries for data from that market are needed, the new server can provide the data, thus reducing data retrieval time.

[0070] At this point, a new server can be purchased, and a new server can be added to the database management system based on the method in the embodiments of this application, and data migration can be performed.

[0071] In one possible implementation, each server in the database management system may include at least one database. Each database may include a category of data. In this embodiment, the data category can be determined based on the market to which the terminal device belongs. For example, a new server may include multiple categories of data. In the configuration information of a new server, the first category identifier may include one or more.

[0072] S202, Based on the configuration information, a database migration task is generated. The database migration task carries a classification identifier of the data to be migrated. The classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated.

[0073] In the initial stage, the new server does not store any data. To reduce the pressure on the database management system by utilizing the new server, data from the global database needs to be migrated to the new server. Therefore, the computer equipment can generate a database migration task based on the configuration information, thereby automatically migrating data and reducing the pressure on the global server.

[0074] The database sharding migration task carries a data classification identifier, which is the first classification identifier in the new server configuration information. Based on this first classification identifier, the data that needs to be migrated to the new server can be found from the global server.

[0075] In one possible implementation, the data to be migrated can be updated to a migration state before data migration, meaning the data can only be queried. When the data is in the migration state, it can only be searched, ensuring that business requests cannot perform other operations on the data during the migration process. This avoids simultaneous data migration and modification of the same data, preventing operational conflicts. Furthermore, if data modification occurs during the migration process, it may be necessary to synchronize the data after the migration is complete. Therefore, this embodiment updates the data to the migration state during the migration process, thus avoiding data inconsistencies after migration and ensuring data accuracy.

[0076] S203, send the classification identifier of the data to be migrated to the data migration service.

[0077] Computer devices can send the classification identifiers of the data to be migrated to the data migration service, enabling the data migration service to perform data migration based on the classification identifiers.

[0078] S204, the data to be migrated from the global server to the newly added server is migrated through the data migration service.

[0079] The data migration service can identify the data to be migrated from the global server; then create a sub-database in the new server; and then copy the data to be migrated from the global server to the sub-database in batches.

[0080] The data migration service can obtain the category identifier of the data to be migrated in the data migration task. Based on the category identifier, the data migration service can find the corresponding first server from the configuration information of the newly added servers in the global database. After determining the first server, the data migration service can create a first database on the first server. The data migration service can then find the data to be migrated from the server according to the first category identifier, and then copy the data to be migrated in batches to the newly created first database.

[0081] For example, the market identifiers received by the data migration service may include 10000, 10001, and 10010. Based on the obtained market identifiers 10000, 10001, and 10010, the data migration service can locate the corresponding target server A from the global server. The data migration service can then create databases 10000, 10001, and 10010 on target server A. For instance, the data migration service can dynamically create database instances ending with the market ID: p_market_api_10000, p_market_api_10001, and p_market_api_10010. The data migration service can retrieve all data with category identifiers 10000, 10001, and 10010 from the global server, and then copy all data with category identifiers 10000, 10001, and 10010 in batches to the corresponding databases p_market_api_10000, p_market_api_10001, and p_market_api_10010.

[0082] Of course, the data migration service can execute migration tasks for multiple new servers simultaneously. For example, the market identifiers received by the data migration service may include 10000, 10001, and 10010. Based on the obtained market identifiers 10000, 10001, and 10010, the data migration service can find the target server A corresponding to 10000 and 10001 from the global server. The data migration service can create databases 10000 and 10001 on target server A. For example, the data migration service can dynamically create library instances p_market_api_10000 and p_market_api_10001 on target server A. The data migration service can find all data with category identifiers 10000 and 10001 from the global server, and then batch copy all data with category identifiers 10000 and 10001 to the corresponding newly created databases p_market_api_10000 and p_market_api_10001. The data migration service can locate the target server B corresponding to category identifier 10010 from the global server based on the obtained market identifier 10010. The data migration service can then create a new database named 10010 on target server B. For example, the data migration service can dynamically create a database instance named p_market_api_10010 on target server B. The data migration service can then locate all data with category identifier 10010 from the global server and batch copy all data with category identifier 10010 to the corresponding database p_market_api_10010 on server B.

[0083] After the data migration task is completed, the corresponding data is stored in the newly added database, and the data stored in the global server has also changed. Therefore, the server configuration mapping information can be updated in the global server. The server configuration mapping information is used to represent the mapping relationship between the server and the stored data; and the updated server configuration mapping information is synchronized to each application service; each application service can respond to received business requests based on the updated server configuration mapping information.

[0084] In this embodiment, data migration can be automatically implemented in the database management system based on the configuration information of the newly added server. This allows for dynamic expansion of the database management system, distributing data across various servers, which then manage the data and respond to user requests, thereby improving the response speed of the database management system.

[0085] The database management system in this embodiment is highly scalable, supporting the addition of new servers and their dynamic allocation to new marketplaces or tenants, thus enabling the expansion and extension of data source resources anytime, anywhere. Furthermore, the database management system can leverage the publish / subscribe features of caching and message middleware to ensure real-time updates of server changes, guaranteeing the operational accuracy of various application services.

[0086] Referring to Figure 3, a flowchart illustrating the steps of another database management method provided in this application embodiment is shown, which may specifically include the following steps:

[0087] S301, if a service request is received from a terminal device, the transaction manager that executes the service request is determined.

[0088] In this embodiment, the execution entity is a computer device. The computer device can deploy client services of a database management system to provide services to terminal devices. For example, it can provide data to terminal devices and update data in the database based on requests from terminal devices.

[0089] The aforementioned service request can be sent from the terminal device to the computer device, which can then retrieve data from the server based on the request. The service request may carry the terminal device's device identifier and a second category identifier, which can be used to determine the storage location of the data required during service execution.

[0090] In this embodiment, the database management system may include a local transaction processor and a cross-service processor. If a service request is received from a terminal device, the transaction manager that executes the service request can be determined first.

[0091] For example, in this embodiment, to support business functions such as registration, import, and migration of terminal devices that require cross-global and sharded databases, distributed transaction Atomikos technology can be introduced into the database management system. This component can be a separate project JAR file. Each backend service application that needs to enable sharding needs to import this component via Maven to enable dynamic server functionality.

[0092] In this embodiment, the transaction processor can be dynamically selected through a custom `@XATransactional` annotation. If the business request data has an `@XATransactional` declaration annotation, it will be processed through XA distributed transactions. XA distributed transactions refer to situations where an operation needs to process two or more different databases simultaneously. Within the same database, transaction integrity can be achieved through the database's own ACID mechanism. However, cross-database operations require a coordinator or mechanism to ensure that both databases either succeed or fail, otherwise data inconsistency will occur. If the business request data does not have an `@XATransactional` annotation, the database management system can automatically use a local transaction manager to execute the business. Local transactions are those that do not require cross-database operations, such as business operations that only update a single global database or a separate database. In this case, a single local transaction can be executed directly, for example, based on the database's own ACID mechanism. ACID refers to the four properties that a database management system must possess to ensure the correctness and reliability of transactions during the writing or updating process: atomicity, consistency, isolation, and durability.

[0093] S302, based on the second category identifier carried in the service request and the server configuration mapping information, determine the second database where the target data is located, wherein the target data is the data required to execute the service request.

[0094] The aforementioned second classification identifier can be a classification identifier for data required during business execution. Computer equipment can obtain server configuration mapping information from the global server. This information includes the mapping relationship between servers and stored data; for example, it may include the association between server identifiers and the classification identifiers of stored data. Therefore, based on the second classification identifier, the corresponding server identifier can be found, thus determining the second server where the target data resides. The second server may include one or more databases. If the server includes only one database, that database is the second database. If the server includes multiple databases, each database can have a database name, which may include classification identifier information, allowing the determination of the second database where the target data resides based on the database name. Alternatively, the server may include the association between the database name of each database and the classification identifier of the stored data, allowing the determination of the second database based on the second classification identifier.

[0095] For example, a business request needs to update data for the European and American markets. The category identifier for this data could be 10000, in which case the second category identifier would be 10000. Assuming that in the server configuration mapping information, server A is associated with category identifiers 10000 and 10001, then based on the server configuration mapping information and the second category identifier 10000, the computer device can determine that the target server is server A. Server A may include databases p_market_api_10000 and p_market_api_10001. Database p_market_api_10000 stores data with category identifier 10000, and database p_market_api_10001 stores data with category identifier 10001. Based on the relationship between the databases and category identifiers in server A, the computer device can determine that the business data with identifier 10000 is stored in database p_market_api_10000 in server A.

[0096] In one possible implementation, the database management system may include code used to perform business operations. In this embodiment, the code for performing business operations can be stored in folders. For each folder, it can be labeled whether it has database sharding characteristics, without needing to be labeled within the code itself. Database sharding characteristics characterize whether the code in a file needs to be executed across different databases. For example, code operating on the same database can be stored in a single folder. The database management system can store a list of relationships between various codes and storage paths, where each code can be associated with the name of the database it operates on. For example, code operating on the global database can be placed in the global folder; code operating on the market sub-database must be placed in the market folder. Most business operations are completed within a single market sub-database, thereby reducing the overhead of distributed transactions.

[0097] Based on the storage path of the target code executing the business request, the storage path contains annotation information that can be used to indicate whether a database sharding operation should be performed. The computer device can determine whether to perform a database sharding operation based on the annotation information of the storage path. If no database sharding operation is performed, the second database can be determined as the global database in the global server; if a database sharding operation is performed, the second database can be determined from the sharding databases in servers outside the global server based on the second category identifier and server configuration mapping information. For example, the computer device may contain a component that can identify whether a database sharding lookup is needed based on the annotation information of the storage path corresponding to the target code executing the business operation. If a database sharding lookup is needed, the second database is found based on the second category identifier and server configuration mapping information. Servers outside the global server can be the aforementioned sharding servers, and each sharding server can include one or more sharding databases. After the data to be migrated is migrated to the new server, the first database on the new server can be a sharding database.

[0098] After the second database is determined, the computer device can extract the target data from the second database.

[0099] S303, Obtain the target data from the second database.

[0100] In this embodiment, it is not necessary to add a database sharding annotation to each database operation code. It is only necessary to configure the package path (relative directory of the file) where the code is located once, thereby ensuring data isolation.

[0101] In this embodiment, the system components can use aspect-oriented programming (AOP) technology to intercept the method calls for processing user request information in the core business logic, extract the identity information of the currently logged-in user or caller from the thread context, extract the second category identifier carried by the terminal device, and then, based on the second category identifier, retrieve the second database of the second server where the target data is located from the server configuration information, and route all subsequent database access object (DAO) layer operations of the database to the second database.

[0102] S304, Execute the service request based on the target data.

[0103] After acquiring the target data, the computer device can perform the corresponding operation based on the business request, such as updating the data according to the business request.

[0104] S305, the execution result of the service request is returned to the terminal device.

[0105] In this embodiment, when responding to business requests from terminal devices, the database management system can determine the transaction manager based on data characteristics, thereby ensuring data consistency within the database management system. In this embodiment, the system automatically identifies whether database sharding is required based on the annotation information of the code's storage path. The necessity of database sharding for code execution can be annotated in the code's storage path, eliminating the need for code annotations in all data access layers. This ensures that during code execution, the system can intelligently determine whether database sharding is required based on the storage path annotation information, and further determine the second database based on the second category identifier. Business operations that do not require database sharding are routed to the global database by default. Directly calling the corresponding code avoids accidental operations on other data, ensuring data isolation and data security.

[0106] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] To better illustrate the solutions in this application, specific embodiments are described below.

[0108] Figure 4 is a sequence diagram of a database management system provided in an embodiment of this application. As shown in Figure 4, after the application service starts, it can automatically assemble and build SpringBeans, construct transactions, transaction managers, and data source loaders. The application service can build a dynamic data source through SPI. Then, the dynamic data source calls the data source dynamic loader to obtain data source configuration information. Specifically, the data source dynamic loader can query data source configuration information from the global database. The global database can return data source configuration information to the data source dynamic loader. After obtaining the data source configuration information, the dynamic data source can load the data source into memory according to the data source configuration information and configure the correspondence between the user marketplace ID and the data source. Here, SPI is a Spring class, and the user marketplace ID is equivalent to the category identifier for the stored data mentioned above.

[0109] The newly added data source in Figure 4 refers to the newly added server mentioned above. When a new data source exists, the user can configure it based on the application service, allowing the application service to obtain the configuration information. The application service can add the configuration information of the new data source to the global database and publish the new data source message using Redis. Redis can then synchronize the updated data source configuration information to each application service. Based on the updated configuration information, the application service can dynamically create the data source and load it into memory for later use. When a user performs a business operation, the application service can dynamically identify the transaction type. Based on the transaction type, the application service can dynamically switch data sources according to the user's marketplace ID. After determining the data source, the application service can dynamically switch databases based on the user's marketplace ID. The application service can intelligently route business operations to the corresponding data source's database, allowing the database to return sharded data to the application service. The application service can then assemble the data based on the sharded data and return it to the user.

[0110] Figure 5 is a flowchart of a database management method provided in an embodiment of this application. As shown in Figure 5, when the application starts, it can read the sub-database configuration information from the main database by default. The main database can be the global database mentioned above, and the sub-database configuration information can be the server configuration mapping information mentioned above. Then, based on the sub-database configuration information, n data sources are created and a market-data source mapping table is constructed. The execution of code methods is intercepted, and the corresponding data source name is automatically selected according to the user market information and whether XA transaction declaration is enabled. In this embodiment, the system components can use AOP technology to intercept the method calls for processing user request information in the core business logic, extract the identity information of the currently logged-in user or caller from the thread context, extract the category identifier information of the terminal device, and then retrieve the server identifier of the destination data source from the server configuration mapping information based on the category identifier information. The data source can be the server mentioned above.

[0111] Based on the data source name, it can be determined whether to enable distributed transactions. If the data source is an XA data source, distributed transactions can be executed; if the data source is a regular data source, local transactions can be executed directly. The target database is then identified, and corresponding operations are performed based on the target database.

[0112] Figure 6 is a flowchart illustrating the addition of a server in a database management system according to an embodiment of this application. In Figure 6, WebApp refers to the backend service of the database management system, such as the backend service application of the maxstore platform, and DataTransfer APP is a data migration service that can be temporarily enabled only during database sharding migration.

[0113] After the data migration is complete, subsequent new versions of the WebApp or each restart will read the latest configuration information from the global database and then dynamically switch to the corresponding data source. Each application service can automatically read the application marketplace data source configuration information from the global database's data source configuration information table (i.e., the server configuration mapping information mentioned above) upon startup, construct XA data sources and ordinary JDBC data sources, load them into memory, and simultaneously build a marketplace data source configuration mapping table.

[0114] As shown in Figure 6, when adding a new data source, the data source management module can create a database migration task for the market. The new data source refers to the newly added server mentioned above, and the data source management module is the server management module. The new data source can be used to store data for a specific market. Platform administrators and operations personnel can configure the data in the new data source. Platform administrators can configure the data source through the data source configuration function in the management backend. After configuration, the market data for the corresponding market will be automatically migrated. During the migration process, the market data being migrated is in a migration state, meaning that the market data can only be viewed and cannot be modified. After the data migration is complete, the new server configuration information officially takes effect, and the database management system can publish a notification of the data source configuration change using Redis's pubSub technology. After receiving the data source configuration change message, each WebAPP service can dynamically construct the data source and load it into memory, while simultaneously updating the market data source configuration mapping table.

[0115] To better illustrate the solution in this application, a specific example is provided below.

[0116] Company A's terminal devices are sold in the first region. Users require platform support when using these devices. Terminal devices can register on the platform, and the data generated by the devices can be stored there. Based on this recorded data, the platform can provide services to the terminal devices.

[0117] In the initial stage, the database management system is used to provide data management services to a limited number of terminal devices. The data generated and the business processes executed by these devices are limited; therefore, a single server can perform all operations. At this point, data management can be based on a single server.

[0118] As the market expands and the number of terminal devices increases, the load on a single server becomes excessive. To improve data response speed, servers can be added to the database management system. Based on steps S201-S204, dynamic server expansion can be automatically implemented, thereby improving data response speed. Subsequently, various application services in the database management system can respond to terminal device requests based on steps S301-S306.

[0119] The method in this embodiment enables dynamic database sharding of the database management system, thereby improving data response speed and ensuring secure isolation of multi-market data when dealing with massive amounts of data. Simultaneously, it solves the problems of automatic market-based database sharding and cross-database query statistics without introducing additional middleware or any intrusiveness.

[0120] Referring to Figure 7, a schematic diagram of a database management device provided in an embodiment of this application is shown, which may specifically include an adding module 71, a generating module 72, a sending module 73, and a migration module 74, wherein:

[0121] Add module 71 is used to add the configuration information of the new server to the global server if a new server is added. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server.

[0122] The generation module 72 is used to generate a database migration task according to the configuration information. The database migration task carries a classification identifier of the data to be migrated. The classification identifier of the data to be migrated is used to indicate the target server to which the data to be migrated is to be migrated.

[0123] Sending module 73 is used to send the classification identifier of the data to be migrated to the data migration service;

[0124] Migration module 74 is used to migrate the data to be migrated from the global server to the newly added server through the data migration service.

[0125] In one possible implementation, the migration module 74 described above includes:

[0126] A determination submodule is used to determine the data to be migrated from the global server through the data migration service;

[0127] Create a submodule for creating the first database in the newly added server;

[0128] The replication submodule is used to copy the data to be migrated from the global server to the first database in batches.

[0129] In one possible implementation, the device further includes:

[0130] The update module is used to update the server configuration mapping information in the global server, wherein the server configuration mapping information is used to represent the mapping relationship between server and tenant data;

[0131] The synchronization module is used to synchronize the updated server configuration mapping information to various application services;

[0132] The response module is used to respond to received business requests based on the updated server configuration mapping information.

[0133] In one possible implementation, the above response module includes:

[0134] The determination submodule is used to determine the transaction manager to execute the service request if a service request is received from a terminal device.

[0135] The response submodule is used to respond to the business request through the transaction manager.

[0136] In one possible implementation, the business request carries a second classification identifier, and the aforementioned response submodule includes:

[0137] The target database determination unit is used to determine the second database where the target data is located based on the second category identifier carried in the business request and the server configuration mapping information, wherein the target data is the data required to execute the business request;

[0138] The acquisition unit is used to acquire the target data from the second database;

[0139] An execution unit is used to execute the business request based on the target data;

[0140] The return unit is used to return the execution result of the service request to the terminal device.

[0141] In one possible implementation, the above-mentioned execution unit includes:

[0142] The judgment subunit is used to determine whether to perform a database sharding operation based on the storage path of the target code that executes the business request.

[0143] The first determining subunit is used to determine that the second database is the global database in the global server if no database sharding operation is performed;

[0144] The second determining subunit is used to determine the second database from the sub-databases in servers other than the global server, based on the second classification identifier and the server configuration mapping information, if a database sharding operation is performed.

[0145] In one possible implementation, the above-mentioned device further includes:

[0146] The suspend module is used to update the data to be migrated to a migration status, where the data can only be queried.

[0147] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0148] Figure 8 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. As shown in Figure 8, the computer device 8 of this embodiment includes: at least one processor 80 (only one is shown in Figure 8), a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80. When the processor 80 executes the computer program 82, it implements the steps in any of the above-described method embodiments.

[0149] The computer device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that FIG8 is merely an example of the computer device 8 and does not constitute a limitation on the computer device 8. It may include more or fewer components than illustrated, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0150] The processor 80 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0151] In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 may be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Furthermore, the memory 81 may include both internal and external storage units of the computer device 8. The memory 81 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0152] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0153] This application provides a computer program product that, when run on a computer device, enables the computer device to perform the steps described in the above-described method embodiments.

[0154] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A database management method, characterized in that, include: If a new server is added, the configuration information of the new server is added to the global server. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server. Based on the configuration information, a database migration task is generated. The database migration task carries a classification identifier of the data to be migrated. The classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated. Send the classification identifier of the data to be migrated to the data migration service; The data migration service is used to migrate the data to be migrated from the global server to the newly added server.

2. The method as described in claim 1, characterized in that, The process of migrating the data to be migrated from the global server to the newly added server using the data migration service includes: The data to be migrated is determined from the global server through the data migration service. Create the first database on the newly added server; The data to be migrated in the global server is copied in batches to the first sub-database.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: Update the server configuration mapping information in the global server. The server configuration mapping information includes the mapping relationship between the server and the stored data. Synchronize the updated server configuration mapping information to all application services; The server responds to the received business requests based on the updated server configuration mapping information.

4. The method as described in claim 3, characterized in that, The process of responding to received service requests based on the updated server configuration mapping information includes: If a service request is received from a terminal device, the transaction manager that executes the service request is determined. The transaction manager responds to the business requests.

5. The method as described in claim 4, characterized in that, The business request carries a second category identifier, and responding to the business request through the transaction manager includes: Based on the second classification identifier and the server configuration mapping information, the second database where the target data is located is determined, and the target data is the data required to execute the business request; Obtain the target data from the second database; Execute the business request based on the target data; The execution result of the service request is returned to the terminal device.

6. The method as described in claim 5, characterized in that, The step of determining the second database where the target data is located based on the second classification identifier and the server configuration mapping information includes: Based on the storage path of the target code that executes the business request, determine whether to perform a database sharding operation; If the database sharding operation is not performed, then the second database is determined to be the global database in the global server; If the database sharding operation is performed, the second database is determined from the sharding databases of servers other than the global server, based on the second category identifier and the server configuration mapping information.

7. The method as described in claim 1, characterized in that, After generating the database sharding migration task based on the configuration information, the method further includes: Update the data to be migrated to a migration status, where the data can only be queried.

8. A database management system, characterized in that, include: The system includes a server cluster, application services, message middleware, data migration services, and data push services. The server cluster includes a global server and sub-servers. The global server stores server configuration mapping information, which includes the mapping relationship between the server and the stored data. The application service is used to add the configuration information of a newly added server to the global server if a new server is added. The configuration information includes a new server identifier and a first category identifier. The new server identifier is used to uniquely identify the new server, and the first category identifier is used to identify the data stored in the new server. Based on the configuration information, a database migration task is generated. The database migration task carries the category identifier of the data to be migrated, and the category identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is to be migrated. Send the classification identifier of the data to be migrated to the data migration service; The data migration service is used to migrate the data to be migrated from the global server to the newly added server; The application service is used to update the server configuration mapping information in the global server after the database migration task is completed, and to call the data push service to synchronize the updated server configuration mapping information to other application services.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images