Database management method, system, equipment and medium

Abstract

The embodiment of the invention is applicable to the technical field of databases, and provides a database management method, system and device and a medium, the method comprises the following steps: if a newly added server exists, adding configuration information of the newly added server to a global server, the configuration information comprising a newly added server identifier and a first classification identifier, the newly-added server identifier is used for uniquely identifying the newly-added server, and the first classification identifier is used for identifying data stored in the newly-added server; according to the configuration information, a sub-library migration task is generated, and the sub-library migration task carries a classification identifier of the to-be-migrated data; sending the classification identifier of the to-be-migrated data to a data migration service; and migrating the to-be-migrated data in the global server to the newly added server through the data migration service. By means of the method, the data can be automatically migrated to the newly-added server, dynamic expansion of the database management system is achieved, and the response speed of the database management system is increased.

Description

Technical Field

[0001] The present application belongs to the field of database technology, and in particular, relates to a database management method, system, device and medium. Background Art

[0002] When the platform provides services to users, it needs to manage data so that it can respond to user requests based on the data. As the number of users using platform services increases, the amount of data that needs to be managed on the platform will also increase.

[0003] Massive amounts of data pose challenges to the platform's database management function. When the platform generates a large amount of data every day, the amount of data the platform searches for increases, resulting in a decrease in data response speed and a reduced user experience. Summary of the invention

[0004] In view of this, embodiments of the present application provide a database management method, system, device and medium to achieve dynamic expansion of a data management server cluster and improve the response speed of the data management server cluster.

[0005] A first aspect of an embodiment of the present application provides a database management method, including:

[0006] If there is a new server, add the configuration information of the new server to the global server, the configuration information including the new server identifier and a first classification identifier, the new server identifier is used to uniquely identify the new server, and the first classification identifier is used to identify the data stored in the new server;

[0007] Generate a sub-database migration task according to the configuration information, wherein the sub-database migration task carries a classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate a first database to which the data to be migrated is migrated;

[0008] Sending the classification identification of the data to be migrated to the data migration service;

[0009] The data to be migrated in the global server is migrated to the newly added server through the data migration service.

[0010] A second aspect of an embodiment of the present application provides a database management system, including: a server cluster, an application service, a message middleware, a data migration service, and a data push service, wherein the server cluster includes a global server and a sub-server, and the global server stores server configuration mapping information, and the server configuration mapping information includes a mapping relationship between a server and stored data;

[0011] The application service is used to add the configuration information of the newly added server to the global server if there is a newly added server, the configuration information including the newly added server identifier and a first classification identifier, the newly added server identifier is used to uniquely identify the newly added server, and the first classification identifier is used to identify the data stored in the newly added server; according to the configuration information, a sub-library migration task is generated, the sub-library migration task carries the classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is migrated; and the classification identifier of the data to be migrated is sent to the data migration service;

[0012] The data migration service is used to migrate the data to be migrated in the global server to the newly added server;

[0013] The application service is used to update the server configuration mapping information in the global server after the sub-database migration task is completed, and call the data push service to synchronize the updated server configuration mapping information to other application services.

[0014] A third aspect of an embodiment of the present application provides a database management device, including:

[0015] An adding module, configured to add configuration information of a newly added server to the global server if there is a newly added server, wherein the configuration information includes a newly added server identifier and a first classification identifier, wherein the newly added server identifier is used to uniquely identify the newly added server, and the first classification identifier is used to identify data stored in the newly added server;

[0016] A generation module, configured to generate a sub-database migration task according to the configuration information, wherein the sub-database migration task carries a classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate a first database to which the data to be migrated is migrated;

[0017] A sending module, used for sending the classification identification of the data to be migrated to the data migration service;

[0018] A migration module is used to migrate the data to be migrated in the global server to the newly added server through the data migration service.

[0019] A fourth aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0020] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0021] A sixth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device executes the method described in the first aspect.

[0022] Compared with the prior art, the embodiments of the present application have the following advantages:

[0023] The database management method in the embodiment of the present application can be applied in a database management system, and the database management system may include a global server. When there is a newly added server in the database management system, the information of the newly added server can be added to the global server, and the configuration information may include the server identifier and classification identifier of the newly added server. Among them, the server identifier can be used to uniquely identify the newly added server. The classification identifier is used to identify the data stored in the newly added server. Based on the configuration information, a sub-library migration task can be generated, so that the corresponding data in the global database can be migrated to the newly added server. When performing sub-library migration, the classification identifier with the migration data can be sent to the data migration service, so that the data to be migrated in the global server can be migrated to the newly added server through the data migration service. In the present application, servers can be dynamically added to the database system, so that the data can be dispersed to each server, and each server manages the data and responds to user needs, thereby improving the response speed of the database management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0025] Figure 1 is a schematic diagram of a database management system provided in an embodiment of the present application;

[0026] Figure 2 It is a schematic diagram of the steps of a database management method provided in an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the steps of another database management method provided by an embodiment of the present application;

[0028] Figure 4 is a timing diagram of a database management system provided in an embodiment of the present application;

[0029] Figure 5is a flow chart of a database management method provided by an embodiment of the present application;

[0030] Figure 6 It is a flowchart of adding a new server in a database management system provided by an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a database management device provided in an embodiment of the present application;

[0032] Figure 8 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0034] The service platform can meet the needs of agents and provide data storage services to multiple connected terminal devices. The service platform can store a large amount of data, making it easier to provide services to terminal devices. As the number of terminal devices connected to the service platform increases, the amount of data it needs to manage will also increase, and the efficiency will decrease when responding to the needs of terminal devices.

[0035] For example, as the number of application markets and agents entering the MAXSTORE platform increases, the number of connected mobile terminal devices also surges. When the number of terminal devices on the MAXSTORE platform reaches tens of millions, each terminal device needs to regularly report data to the platform and receive task instructions issued by the platform. The platform generates billions of data every day. How to quickly retrieve data and respond to terminal requests under the premise of large data volume and high concurrency and ensure data isolation and security has become a big challenge.

[0036] The open source solutions for dynamic database partitioning on the market include ShardingJDBC and baomidou-dynamic-datasource. Both solutions are based on the annotation aspect method, which requires adding annotations to all database operations, and are relatively invasive. In addition, ShardingJDBC configuration is relatively complex, and the partitioned databases are mostly completely equivalent. The MAXSTORE platform library is a model of 1 global library (global) + n sub-markets (market), and each sub-market library needs to interact with the global library. Therefore, it involves cross-library operation scenarios, that is, the solution needs to support distributed transactions. However, baomidou-dynamic-datasource conflicts with Spring native transactions in supporting distributed transactions and cannot coexist, and MAXSTORE's master-slave database selection is based on Spring native read-only transactions.

[0037] Based on this, the embodiments of the present application provide a database management system and a database management method for improving the response efficiency of the service platform and ensuring data isolation security.

[0038] The technical solution of the present application is described below through specific embodiments.

[0039] The present embodiment provides a database management system, which may include: a server cluster, an application service, a message middleware, a data migration service, and a data push service, wherein the server cluster may include a global server and a sub-server, and the global server may store server configuration mapping information, and the server configuration mapping information may include a mapping relationship between the server and the stored data. Each server in the server cluster may have a unique server identifier, and the server identifier may be used to uniquely identify the server. Each classification identifier may correspond to a data set. The server configuration mapping information may be an association relationship between the server identifier of the server and the classification identifier of the stored data. It should be noted that each server identifier may be associated with one or more classification identifiers.

[0040] The above-mentioned application service can be used to add the configuration information of the newly added server to the global server if there is a newly added server, the configuration information includes the newly added server identifier and the first classification identifier, the newly added server identifier is used to uniquely identify the newly added server, and the first classification identifier is used to identify the data stored in the newly added server; according to the configuration information, generate a sub-database migration task, the sub-database migration task carries the classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is migrated; send the classification identifier of the data to be migrated to the data migration service;

[0041] The above data migration service can be used to migrate the data to be migrated in the global server to the newly added server;

[0042] The above application service can also be used to update the server configuration mapping information in the global server after the sub-database migration task is completed, and call the data push service to synchronize the updated server configuration mapping information to other application services.

[0043] Figure 1 It is a schematic diagram of a database management system provided in an embodiment of the present application. Figure 1 In the database management system shown, the server cluster may include MYSQL global and MYSQL market. MYSQL global is a global database, which may be set in the above-mentioned global server. The global database may include the registration information of each terminal device. When the terminal device registers and modifies the registration information, the data needs to be synchronized in the global database.

[0044] MYSQL market is a market database, that is, a sub-database, which can exist in a server other than the global server. The data stored in the sub-database can be the relevant data of the terminal devices in a market area. The market area can be divided according to geographical areas or according to a set method. Each market area can have a market identifier.

[0045] Exemplarily, the terminal device may be a POS machine, and based on the sales area of ​​the POS machine, the domestic market, the North American market, the European market, etc. The registration information of each POS machine may be stored in the global database, and the data of each market may be divided into sub-databases.

[0046] When a database management system starts to provide services, generally, the number of connected terminal devices is small, so the database management system does not need to manage much data. At this time, the database management system may include only one server. The server may be a global server, based on which the database management system can receive data from various terminal devices and respond to requests from various terminal devices.

[0047] As the number of terminal devices connected to the database management system increases, the data that needs to be managed also gradually increases. A large amount of data is concentrated in the global database, which will reduce the efficiency of data search. At the same time, the concentration of data also brings challenges to data security.

[0048] At this time, a server may be added to the database management system so that the newly added server can provide services to a portion of the terminal devices.

[0049] When a new server is added to the database management system, the configuration information of the new server can be obtained. The configuration information can be set by the administrator. The configuration information may include the new server identifier of the new server and the first classification identifier of the data stored in the new server. The above-mentioned new server identifier can be used to uniquely identify the new server. The first classification identifier may correspond to a data set. Exemplarily, the first classification identifier may be a market identifier, and the market identifier may be used to identify the market to which the terminal device is divided. For example, the market identifier may include 1, 2, 3, ···, n, where 1 can be used to identify the domestic market, 2 can be used to identify the North American market, 3 can be used to identify the European and American markets, etc. Each piece of data stored in the server may include a market identifier to identify the market to which the data corresponds.

[0050] Based on the configuration information of the newly added server, a sub-database migration task can be generated, which can be used to migrate data from the global database to the newly added server. The classification identifier of the data to be migrated carried by the sub-database migration task can be the first classification identifier of the data to be stored in the newly added server.

[0051] The database management system can perform data migration through the data migration service. The classification identification of the data to be migrated can be carried in the sub-database migration task, and the data migration service can obtain the classification identification of the data to be migrated in the sub-database migration task; based on the classification identification of the data to be migrated, the data migration service can search for the corresponding first server from the server configuration mapping information in the global database, and the first server is the server that stores the data to be migrated after the data migration is completed. After determining the first server, the data migration service can create a first database in the first server. The data migration service can search for the data to be migrated from the global server according to the classification identification of the data to be migrated, and then batch copy the data to be migrated to the newly created first database in the first server. Among them, the first server can be a newly added server.

[0052] It should be noted that when executing a sub-database migration task, the data corresponding to the sub-database migration task can be suspended. That is, during the execution of the sub-database migration task, the data to be migrated can only be searched, and data cannot be added, modified, or deleted. During the execution of the sub-database migration task, suspending the data to be migrated can avoid data confusion during the data migration process.

[0053] The server configuration mapping information may include the mapping relationship between the server and the stored data. After the data migration is completed, the data storage location changes, and the mapping relationship between the server and the stored data will inevitably change. Therefore, after the sub-library migration task is completed, the server configuration mapping information in the global server can be updated. The server configuration mapping information may include the mapping relationship between the server and the stored data, and specifically may include the association relationship between the server ID of the server and the classification ID of the stored data. This is equivalent to, after the data migration is completed based on the configuration information, the association relationship between the server ID and the classification ID can be updated according to the results of the data migration, thereby completing the update of the server configuration mapping information.

[0054] After the server configuration mapping information is updated, the database system can synchronize the server configuration mapping information to all application services in the data system through the data synchronization service, which is equivalent to notifying each application service to provide services to the terminal device based on the new server configuration mapping information.

[0055] In one possible implementation, the data in the newly added server may include a classification identifier. Exemplarily, the configuration information of the newly added server may include server identifier a and classification identifier 2, and the configuration information may be stored in the global server. The data migration service may obtain classification identifier 2 of the sub-database migration task, and based on classification identifier 2, the data migration service may determine the target server a. The data migration service may create a new database a in the target server a. The data migration service may search for all data with classification identifier 2 from the global server, and then batch copy all data with classification identifier 2 to the newly created database a, thereby completing the sub-database migration task. After the migration is completed, the association between the global server and classification identifier 2 in the server configuration information may be deleted; at the same time, the association between server identifier a and classification identifier 2 may be added to the server configuration mapping information.

[0056] In another possible implementation, the data in the newly added server may include multiple classification identifiers. Exemplarily, the configuration information of the newly added server may include server identifier a and classification identifiers 2 and 3, and the configuration information may be stored in the global server. The data migration service may obtain classification identifiers 2 and 3 of the sub-database migration task, and the data migration service may determine the target server a based on classification identifiers 2 and 3. The data migration service may create a new database a and a database b in the target server a. The data migration service may search for all data with classification identifier 2 from the global server, and then batch copy all data with classification identifier 2 to the newly created database a; search for all data with classification identifier 3 from the global server, and then batch copy all data with classification identifier 3 to the newly created database b, thereby completing the sub-database migration task. After the migration is completed, the association between the global server and classification identifier 2 and classification identifier 3 in the server configuration information may be deleted; at the same time, the association between server identifier a and classification identifier 2 and classification identifier 3 may be added to the server configuration mapping information.

[0057] Each application service can respond to the request of the terminal device based on the updated server configuration mapping information. The application service can find the server where the data required to respond to the request is located according to the terminal device identifier and the server configuration mapping information, and find the corresponding data from the server's target database.

[0058] In a database management system, new servers can be added to a server cluster to automatically migrate data. This embodiment can dynamically add servers to the database system to distribute data to each server, which can then manage the data and respond to user needs, thereby improving the response speed of the database management system.

[0059] Reference Figure 2 , a schematic diagram of a step flow of a database management method provided in an embodiment of the present application may specifically include the following steps:

[0060] S201, if there is a new server, add the configuration information of the new server to the global server, the configuration information includes the new server identifier and a first classification identifier, the new server identifier is used to uniquely identify the new server, and the first classification identifier is used to identify the data stored in the new server.

[0061] The execution subject of this embodiment may be a computer device, which may be a client device of a server cluster. One or more application services may be deployed in the computer device, and based on the application services, the database management system may send data to each terminal device and receive service requests from the terminal device.

[0062] In order to reduce the pressure on the database management system, a server can be added to the database system. The above-mentioned newly added server is a server newly added to the database management system. The server does not contain data at the beginning. The administrator can configure the server to determine the data that can be stored and managed in the server.

[0063] The above configuration information can be obtained according to the settings of the administrator. Exemplarily, the computer device can provide a configuration interface, and the administrator can set the newly added server through the configuration interface. The computer device can obtain the administrator's setting data to generate the configuration information of the newly added server. For example, the computer device can deploy a server management module. When there is a new server, the server management module of the computer device can display a configuration interface to the user. The administrator can select or fill in the configuration information of the newly added server in the configuration interface. Based on the information filled in by the administrator, the computer device can automatically obtain the configuration information of the newly added server. When configuring a new server, the administrator can set a server identifier for the new server. The server identifier is the above-mentioned new server identifier, which can be used to uniquely identify the new server. The first classification identifier of the data stored in the new server can also be configured. The first classification identifier can be used to identify the category corresponding to the data stored in the new server. The category can be determined according to the market to which the terminal device corresponding to the data belongs, that is, the first classification identifier can be a market identifier.

[0064] After obtaining the configuration information of the newly added server, the computer device can store the configuration information in the global database, thereby facilitating subsequent data migration.

[0065] It should be noted that, in the embodiment of the present application, each terminal device is registered in the global server, and the update of the registration information of the terminal device also needs to be updated in the global server.

[0066] For example, the terminal device may be a POS machine. When the number of POS machines is small, all data can be stored in a global server, so that the global server can manage and respond to requests in a unified manner. However, as the sales market expands, the number of terminal devices will increase.

[0067] There may be multiple terminal devices in different market areas. The data in multiple markets are concentrated in one server. When searching and responding to data, the server will take longer. At this time, the centralized storage of data affects the service efficiency.

[0068] The account data of different markets are not highly correlated. For example, the business data of POS machines in the North American market and the European market are generally not highly correlated. Therefore, the data of different markets can be stored separately so that different servers can provide services for them. When a new server is added to the server cluster, the data of one of the markets can be transferred to the new server, so that a database can be established for the market in the new server. When the data of the market needs to be queried, the new server can provide the data, thus reducing the time for data search.

[0069] At this time, a new server can be purchased, and a new server can be added to the database management system based on the method in the embodiment of the present application, and data migration can be performed.

[0070] In a possible implementation, each server in the database management system may include at least one database. Each database may include a category of data. In this embodiment, the category of data may be determined according to the market to which the terminal device belongs. For example, a newly added server may include multiple categories of data. In the configuration information of the newly added server, the first classification identifier may include one or more.

[0071] S202: Generate a sub-database migration task according to the configuration information, wherein the sub-database migration task carries a classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate a first database to which the data to be migrated is migrated.

[0072] In the initial stage, the newly added server does not store data. In order to use the newly added server to reduce the pressure on the database management system, the data in the global database needs to be migrated to the newly added server. Therefore, the computer device can generate a sub-library migration task based on the configuration information, so that data migration can be automatically realized based on the sub-library migration task, reducing the pressure on the global server.

[0073] The sub-database migration task carries a classification identifier of the data, which is the first classification identifier in the configuration information of the newly added server. Based on the first classification identifier, the data that needs to be migrated to the newly added server can be found from the global server.

[0074] In a possible implementation, before data migration is performed, the data to be migrated can be updated to a migration state, where the data can only be queried. When the data to be migrated is in the migration state, it can only be searched, which can ensure that business requests cannot perform other operations on the data to be migrated during the migration process, thereby avoiding data migration and data modification on the same data at the same time, avoiding operation conflicts. At the same time, if data modification is performed during the data migration process, it may be necessary to synchronize and update the data after the data migration is completed. Therefore, this embodiment can update the data to be migrated to the migration state during the data migration process, thereby avoiding data synchronization problems after data migration and ensuring the correctness of the data.

[0075] S203: Send the classification identification of the data to be migrated to a data migration service.

[0076] The computer device may send the classification identification of the data to be migrated to the data migration service, so that the data migration service may perform data migration based on the classification identification.

[0077] S204: Migrate the data to be migrated in the global server to the newly added server through the data migration service.

[0078] Through the data migration service, you can determine the data to be migrated from the global server; then create a sub-database in the newly added server; and batch copy the data to be migrated from the global server to the sub-database.

[0079] The data migration service can obtain the classification identifier of the data to be migrated in the data migration task. Based on the classification identifier of the data to be migrated, the data migration service can search for the corresponding first server from the configuration information of the newly added server in the global database. After determining the first server, the data migration service can create a first database in the first server. The data migration service can search for the data to be migrated from the server according to the first classification identifier, and then copy the data to be migrated in batches to the newly created first database.

[0080] Exemplarily, the market identifiers received by the data migration service may include: 10000, 10001, and 10010. The data migration service may search for the corresponding target server A from the global server according to the obtained market identifiers 10000, 10001, and 10010. The data migration service may create new databases 10000, 10001, and 10010 in the target server A. For example, the data migration service may dynamically create library instances ending with market id: p_market_api_10000, p_market_api_10001, and p_market_api_10010. The data migration service can search for all data with category identifiers 10000, 10001, and 10010 from the global server, and then batch copy all data with category identifiers 10000, 10001, and 10010 to the corresponding databases p_market_api_10000, p_market_api_10001, and p_market_api_10010.

[0081] Of course, the data migration service can perform migration tasks for multiple newly added servers at the same time. For example, the market identifiers received by the data migration service may include: 10000, 10001, and 10010. The data migration service can find the corresponding target server A corresponding to 10000 and 10001 from the global server based on the obtained market identifiers 10000, 10001, and 10010. The data migration service can create new databases 10000 and 10001 in the target server A. For example, the data migration service can dynamically create library instances: p_market_api_10000 and p_market_api_10001 in the target server A. The data migration service can find all data with classification identifiers 10000 and 10001 from the global server, and then batch copy all data with classification identifiers 10000 and 10001 to the corresponding newly created databases p_market_api_10000 and p_market_api_10001. The data migration service can find the target server B corresponding to the corresponding market identifier 10010 from the global server according to the obtained market identifier 10010. The data migration service can create a new database 10010 in the target server B. For example, the data migration service can dynamically create a library instance: p_market_api_10010 in the target server B. The data migration service can find all the data with the classification identifier 10010 from the global server, and then batch copy all the data with the classification identifier 10010 to the database p_market_api_10010 of the corresponding server B.

[0082] After the data migration task is completed, the corresponding data is stored in the newly added database, and the data stored in the global server is also changed. Therefore, the server configuration mapping information can be updated in the global server, and the server configuration mapping information is used to represent the mapping relationship between the server and the stored data; and the updated server configuration mapping information is synchronized to each application service; each application service can respond to the received business request based on the updated server configuration mapping information.

[0083] In this embodiment, data migration can be automatically implemented in the database management system based on the configuration information of the newly added server, so that the database management system can be dynamically expanded and the data can be distributed to various servers. Each server will manage the data and respond to user needs, thereby improving the response speed of the database management system.

[0084] The database management system in this embodiment has high scalability, supports adding new servers in the database management system, and dynamically allocates the new servers to new markets or new tenants, so that data source resources can be expanded and extended anytime and anywhere. At the same time, the database management system can also realize the real-time nature of server change information based on the publishing and subscription characteristics of the cache and message middleware, ensuring the accuracy of the operation of each application service.

[0085] Reference Figure 3 , shows a schematic diagram of the steps of another database management method provided in an embodiment of the present application, which may specifically include the following steps:

[0086] S301: If a service request is received from a terminal device, determine a transaction manager for executing the service request.

[0087] The execution subject of this embodiment is a computer device, which can deploy the client service of the database management system to provide services to the terminal device. For example, data can be provided to the terminal device, and data in the database can be updated based on the request of the terminal device.

[0088] The above service request can be sent from the terminal device to the computer device, and the computer device can obtain data from the server according to the service request. The service request can carry the device identification of the terminal device and the second classification identification, and the second classification identification can be used to determine the storage location of the data required during the service execution process.

[0089] In this embodiment, the database management system may include a local transaction processor and a cross-service processor. If a service request is received from a terminal device, a transaction manager for executing the service request may be determined first.

[0090] For example, in this embodiment, in order to support business functions such as registration, import, and migration of terminal devices that need to cross the global database and the sub-database, the distributed transaction Atomikos technology can be introduced in the database management system. This component can be a separate project jar, and each back-end service application that needs to enable the sub-database needs to introduce this component through Maven to enable the dynamic server function.

[0091] In this embodiment, dynamic selection of transaction processors can be achieved by customizing @XATransactional annotations. If the business request data has @XATransactional declaration annotations, it is processed through XA distributed transactions. XA distributed transactions refer to situations where an operation needs to process more than two different databases at the same time. The same database can achieve transaction integrity through the database's own ACID mechanism, but when operating across databases, a coordinator or a mechanism needs to be introduced to ensure that both databases are either successful or failed, otherwise data inconsistency will occur. If the business request data does not have @XATransactional annotations, the database management system can automatically use the local transaction manager to execute the business. Local transactions do not require cross-database situations. For example, business operations will only be updated in a single global database or sub-database. At this time, it can be directly executed through a single local transaction, for example, it can be executed based on the database's own ACID mechanism. Among them, ACID refers to the four characteristics that the database management system must have in order to ensure that the transaction is correct and reliable during the process of writing or updating data: atomicity, consistency, isolation, and persistence.

[0092] S302: Determine a second database where target data is located according to the second classification identifier carried in the service request and the server configuration mapping information, where the target data is data required to execute the service request.

[0093] The above-mentioned second classification identifier can be a classification identifier of data required in the process of business execution. The computer device can obtain server configuration mapping information from the global server, and the server configuration mapping information includes a mapping relationship between the server and the stored data. For example, the server configuration mapping information may include an association relationship between the server identifier and the classification identifier of the stored data. Therefore, according to the second classification identifier, the corresponding server identifier can be found, thereby determining the second server where the target data is located. The second server may include one or more databases. If the server includes only one database, the database is the second database. If the server includes multiple databases, each database may have a database name, and the database name may include classification identification information, so that the second database where the target data is located can be determined based on the database name. Alternatively, the server may include an association relationship between the database name of each database and the classification identifier of the stored data, so that the second database can be determined based on the second classification identifier.

[0094] Exemplarily, the business request needs to update the data of the European and American markets. The classification identifier of the data of the European and American markets can be 10000, and the second classification identifier is 10000. Assuming that in the server configuration mapping information, the classification identifiers associated with server A are 10000 and 10001, then according to the server configuration mapping information and the second classification identifier 10000, the computer device can determine that the target server is server A. Server A may include databases p_market_api_10000 and p_market_api_10001. Database p_market_api_10000 is used to store data with a classification identifier of 10000, and database p_market_api_10001 is used to store data with a classification identifier of 10001. Based on the relationship between the database in server A and the classification identifier, the computer device can determine that the data with a business data identifier of 10000 is stored in the database p_market_api_10000 in server A.

[0095] In one possible implementation, the database management system may include code, and the code is used to perform business operations. In an embodiment of the present application, the code for performing business operations can be stored in folders. For folders, it can be marked whether they have sub-library operation characteristics, and there is no need to mark them inside the code. The sub-library operation characteristics are used to characterize whether the code in the file needs to be executed by the data in the sub-database. Exemplarily, the code for operating the same database can be stored in a folder. The database management system can store a list of relationships for storing each code and storage path, and each code can be associated with the name of the database to be operated. For example, the code for operating the global global library can be placed in the global folder; the code for operating the market sub-library must be placed in the market file. Most business operations are completed under a single market sub-library, thereby reducing the cost of distributed transactions.

[0096] According to the storage path of the target code for executing the business request, the storage path has annotation information, and the annotation information can be used to indicate whether to perform the sub-library operation. The computer device can determine whether to perform the sub-library operation according to the annotation information of the storage path. If the sub-library operation is not performed, it can be determined that the second database is the global database in the global server; if the sub-library operation is performed, the second database can be determined from the sub-library in the server other than the global server according to the second classification identifier and the server configuration mapping information. For example, there may be a component in the computer device, which can identify whether a sub-library search is required according to the storage path information of the target code for executing the business operation. If a sub-library search is required, the second database is found according to the second classification identifier and the server configuration mapping information. The server other than the global server can be the above-mentioned sub-server, and each sub-server can include one or more sub-databases. Among them, after the data to be migrated is migrated to the newly added server, the first database on the newly added server can be a sub-database.

[0097] After determining the second database, the computer device may extract the target data from the second database.

[0098] S303: Acquire the target data from the second database.

[0099] In this embodiment, there is no need to add a sub-library annotation to each database operation code. It only needs to be configured once to the package path (relative file directory) where the code is located, thereby ensuring data isolation.

[0100] In this embodiment, the system component can use aspect oriented programming (AOP) technology to intercept method calls of the core business logic to process user request information, extract the identity information of the current login person or caller from the thread context, extract the second classification identifier carried by the terminal device, and then based on the second classification identifier, retrieve the second database of the second server where the target data is located from the server configuration information, and route all subsequent database access (Data Access Object, DAO) layer operations to the second database.

[0101] S304: Execute the service request based on the target data.

[0102] After acquiring the target data, the computer device can perform operations corresponding to the business request according to the target data, for example, it can update the data according to the business request.

[0103] S305: Return the execution result of the service request to the terminal device.

[0104] In this embodiment, when responding to the service request of the terminal device, the database management system can determine the transaction manager according to the data characteristics, thereby ensuring the data consistency in the database management system. In this embodiment, it is automatically identified whether a sub-library operation is required based on the storage path of the code. The necessity of the sub-library operation of the execution code can be annotated in the storage path of the code, without the need for code annotation in all data access layers, thereby ensuring that when the code is executed, it can intelligently determine whether a sub-library query is required based on the storage path, and then the second database can be further determined based on the second classification identifier, and business operations that do not require sub-library are routed to the global database by default. Directly calling the corresponding code avoids misoperation of other data, ensures data isolation, and ensures data security.

[0105] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0106] In order to better illustrate the solution in this application, specific embodiments are described below.

[0107] Figure 4 is a timing diagram of a database management system provided by an embodiment of the present application. Figure 4As shown, after the application service is started, it can automatically assemble and build SpringBean, and build transactions, transaction managers, and data source loaders. The application service can build a dynamic data source through SPI. After that, the dynamic data source calls the data source dynamic loader to obtain the data source configuration information. Specifically, the data source dynamic loader can query the data source configuration information from the global database. The global database can return the data source configuration information to the data source loader. After the data source loader obtains the data source configuration information, the dynamic data source can load the data source into the memory according to the data source configuration information, and configure the corresponding relationship between the user market id and the data source. Among them, SPI is a class of Spring, and the user market id is equivalent to the classification identifier of the above-mentioned storage data.

[0108] Figure 4 The newly added data source in is the newly added server mentioned above. When there is a new data source, the user can configure the new data source based on the application service, so that the application service can obtain the configuration information of the new data source. The application service can add the configuration information of the new data source to the global database and use Redis to publish the new data source message; Redis can synchronize the updated data source configuration information to each application service. Based on the updated configuration information, the application service can dynamically create a data source and load it into the memory for subsequent use; when the user performs a business operation, the application service can dynamically identify the transaction type; based on the transaction type, the application service can dynamically switch the data source according to the user market id; after determining the data source, the application service can dynamically switch the database according to the user market id; the application service can intelligently route the business operation to the database of the corresponding data source, so that the database can return the sharded data to the application service. The application service can assemble data based on the sharded data and return it to the user.

[0109] Figure 5 1 is a flow chart of a database management method provided by an embodiment of the present application. Figure 5 As shown, when the application starts, the application can read the sub-library configuration information from the main database by default. The main database can be the global database mentioned above, and the sub-library configuration information can be the server configuration mapping information mentioned above; then, based on the sub-library configuration information, n data sources are created and a market and data source mapping table is constructed; the code method execution is intercepted, and the corresponding data source name is automatically selected according to the user market information and whether the XA transaction statement is enabled. In this embodiment, the system component can use AOP technology to intercept the method call of the core business logic to process user request information, extract the identity information of the current login person or caller from the thread context, extract the classification identification information to which the terminal device belongs, and then retrieve the server identification of the destination data source from the server configuration mapping information based on the classification identification information. The data source can be the above-mentioned server.

[0110] Based on the data source name, you can determine whether to enable distributed transactions. If the data source is an XA data source, you can execute distributed transactions; if the data source is a normal data source, you can directly execute local transactions. Hit the target database and perform corresponding operations based on the target database.

[0111] Figure 6 This is a flowchart of adding a new server in a database management system provided by an embodiment of the present application. Figure 6 In the example, WebApp refers to the backend service of the database management system, such as the backend service application of the maxstore platform. DataTransfer APP is a data migration service that can be temporarily enabled only during database migration.

[0112] After the data migration is completed, the subsequent new versions of WebApp or each restart will read the latest configuration information from the global database, and then dynamically switch to the corresponding data source. When each application service is started, it can automatically read the application market data source configuration information from the data source configuration information table of the global database (that is, the above-mentioned server configuration mapping information) and build XA data sources and ordinary JDBC data sources and load them into memory, and build a market data source configuration mapping table.

[0113] like Figure 6 As shown in the figure, when a new data source is added, the data source management module can create a sub-database migration task for the market. Among them, the new data source is the above-mentioned new server, and the data source management module is the server management module. The new data source can be used to store the data of a certain market. The platform administrator and operation and maintenance personnel can initiate the configuration of the data in the new data source. The platform administrator can configure the data source through the management background data source configuration function. After the configuration is completed, the market data of the corresponding market is automatically migrated. The market data that is being migrated during the migration process is in the migration state, that is, the market data can only be viewed and cannot be changed. After the data migration is completed, the new server configuration information is officially effective, and the database management system can publish a message notification of the data source configuration change through Redis's pubSub technology. After receiving the data source configuration change message, each WebAPP service can dynamically build the data source and load it into the memory, and update the market data source configuration mapping table at the same time.

[0114] In order to better illustrate the solution in the present application, the solution in the present application is described below with a specific example.

[0115] Company A's terminal devices are sold in the first region. When users use the terminal devices, they need support from the platform. The terminal devices can be registered on the platform, the data generated by the terminal devices can be stored on the platform, and the platform can provide services to the terminal devices based on the recorded data.

[0116] In the initial stage, the database management system is used to provide data management services to the first number of terminal devices. The data generated by the first number of terminal devices is limited, and the services performed are also limited. At this time, one server can perform all operations. At this time, data management can be performed based on one server.

[0117] As the market expands, the number of terminal devices increases, and the pressure on a server is too great. In order to improve the data response speed, servers can be added to the database management system. Based on the steps S201-S204, the dynamic expansion of the server can be automatically realized, thereby improving the data response speed. Afterwards, each application service in the database management system can respond to the request of the terminal device based on the steps S301-S306.

[0118] The method in this embodiment can realize dynamic database partitioning of the database management system, thereby improving the data response speed when the data volume is huge and ensuring the safe isolation of multi-market data. At the same time, it can solve the problem of automatic market-based database partitioning and cross-database query statistics without introducing additional middleware and any intrusion.

[0119] Reference Figure 7 , shows a schematic diagram of a database management device provided in an embodiment of the present application, which may specifically include an adding module 71, a generating module 72, a sending module 73 and a migration module 74, wherein:

[0120] An adding module 71 is used to add the configuration information of the newly added server to the global server if there is a newly added server, wherein the configuration information includes a newly added server identifier and a first classification identifier, wherein the newly added server identifier is used to uniquely identify the newly added server, and the first classification identifier is used to identify data stored in the newly added server;

[0121] A generating module 72, configured to generate a sub-database migration task according to the configuration information, wherein the sub-database migration task carries a classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate a target server to which the data to be migrated is migrated;

[0122] A sending module 73, used for sending the classification identification of the data to be migrated to the data migration service;

[0123] The migration module 74 is used to migrate the data to be migrated in the global server to the newly added server through the data migration service.

[0124] In a possible implementation, the migration module 74 includes:

[0125] A determination submodule, configured to determine the data to be migrated from the global server through the data migration service;

[0126] A creation submodule, used to create the first database in the newly added server;

[0127] The replication submodule is used to batch-copy the data to be migrated in the global server to the first database.

[0128] In a possible implementation manner, the device further includes:

[0129] An updating module, used to update server configuration mapping information in the global server, wherein the server configuration mapping information is used to characterize a mapping relationship between the server and tenant data;

[0130] A synchronization module, used to synchronize the updated server configuration mapping information to each application service;

[0131] A response module is used to respond to the received service request based on the updated server configuration mapping information.

[0132] In a possible implementation, the response module includes:

[0133] A determination submodule, configured to determine a transaction manager for executing a service request if a service request is received from a terminal device;

[0134] The response submodule is used to respond to the service request through the transaction manager.

[0135] In a possible implementation, the service request carries a second classification identifier, and the response submodule includes:

[0136] a target database determining unit, configured to determine a second database where target data is located according to the second classification identifier carried in the service request and the server configuration mapping information, wherein the target data is data required to execute the service request;

[0137] an acquiring unit, configured to acquire the target data from the second database;

[0138] An execution unit, configured to execute the service request based on the target data;

[0139] A returning unit is used to return the execution result of the service request to the terminal device.

[0140] In a possible implementation, the execution unit includes:

[0141] A judgment subunit, used to determine whether to perform a library division operation according to a storage path of a target code for executing the service request;

[0142] a first determining subunit, configured to determine that the second database is a global database in the global server if the database division operation is not performed;

[0143] The second determining subunit is used to determine the second database from the sub-databases in the server other than the global server according to the second classification identifier and the server configuration mapping information if a sub-database operation is performed.

[0144] In a possible implementation, the above device further includes:

[0145] The suspension module is used to update the data to be migrated to a migration state, where the migration state is a state in which the data can only be queried.

[0146] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0147] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 8 As shown, the computer device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one is shown in the figure), a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, wherein the processor 80 implements the steps of any of the above-mentioned method embodiments when executing the computer program 82.

[0148] The computer device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the computer device 8 and does not constitute a limitation on the computer device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0149] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0150] In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 8. Further, the memory 81 may also include both an internal storage unit of the computer device 8 and an external storage device. The memory 81 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0151] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0152] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer device.

[0153] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A database management method, characterized in that: include: If there is a new server, add the configuration information of the new server to the global server, the configuration information including the new server identifier and a first classification identifier, the new server identifier is used to uniquely identify the new server, and the first classification identifier is used to identify the data stored in the new server; Generate a sub-database migration task according to the configuration information, wherein the sub-database migration task carries a classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate a first database to which the data to be migrated is migrated; Sending the classification identification of the data to be migrated to the data migration service; The data to be migrated in the global server is migrated to the newly added server through the data migration service.

2. The method according to claim 1, characterized in that The migrating the data to be migrated in the global server to the newly added server by using the data migration service includes: Determining the data to be migrated from the global server through the data migration service; Creating the first database in the newly added server; The data to be migrated in the global server is copied in batches to the first database.

3. The method according to claim 1 or 2, characterized in that The method further comprises: Updating server configuration mapping information in the global server, wherein the server configuration mapping information includes a mapping relationship between a server and storage data; Synchronizing the updated server configuration mapping information to each application service; The received service request is responded to based on the updated server configuration mapping information.

4. The method according to claim 3, characterized in that The responding to the received service request based on the updated server configuration mapping information includes: If a service request is received from a terminal device, determining a transaction manager to execute the service request; The transaction manager responds to the service request.

5. The method according to claim 4, characterized in that The service request carries a second classification identifier, and responding to the service request through the transaction manager includes: Determine, according to the second classification identifier and the server configuration mapping information, a second database where target data is located, wherein the target data is data required to execute the service request; Acquire the target data from the second database; Execute the service request based on the target data; The execution result of the service request is returned to the terminal device.

6. The method according to claim 5, characterized in that The step of determining the second database where the target data is located according to the second classification identifier and the server configuration mapping information includes: Determining whether to perform a database partitioning operation according to a storage path of a target code for executing the service request; If the database splitting operation is not performed, determining that the second database is a global database in the global server; If a database division operation is performed, the second database is determined from the database divisions in the server other than the global server according to the second classification identifier and the server configuration mapping information.

7. The method according to claim 5, characterized in that After generating the sub-database migration task according to the configuration information, the method further includes: The data to be migrated is updated to a migration state, where the migration state is a state where the data can only be queried.

8. A database management system, characterized in that: include: Server cluster, application service, message middleware, data migration service and data push service, the server cluster includes a global server and sub-servers, the global server stores server configuration mapping information, and the server configuration mapping information includes a mapping relationship between the server and the stored data; The application service is used to add the configuration information of the newly added server to the global server if there is a newly added server, the configuration information including the newly added server identifier and a first classification identifier, the newly added server identifier is used to uniquely identify the newly added server, and the first classification identifier is used to identify the data stored in the newly added server; according to the configuration information, a sub-library migration task is generated, the sub-library migration task carries the classification identifier of the data to be migrated, and the classification identifier of the data to be migrated is used to indicate the first database to which the data to be migrated is migrated; Sending the classification identification of the data to be migrated to the data migration service; The data migration service is used to migrate the data to be migrated in the global server to the newly added server; The application service is used to update the server configuration mapping information in the global server after the sub-database migration task is completed, and call the data push service to synchronize the updated server configuration mapping information to other application services.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images