Data storage scheduling management system, application method and device

Abstract

The invention relates to the field of data storage management, and discloses a data storage scheduling management system and an application method and device.According to the technical scheme, the data storage scheduling management system comprises a scheduling manager used for logic management; the annotation analyzer is used for scanning annotations marked by the logic table, determining a data storage mode and constructing a data storage mapping cache of the logic table; the configuration register is used for constructing parameter configuration of a horizontal fragmentation storage mode and a vertical sub-library storage mode; the data source manager is used for creating a connection pool component corresponding to each data storage mode; the dynamic routing manager is used for registering data storage mapping, parameter configuration and core information of the connection pool component, and is also used for judging and routing a data storage mode when the application system runs; and the data fragmentation collector is used for horizontally fragmenting the data in the single-library single-table storage mode.

Description

Technical Field

[0001] The present invention relates to the field of data storage management, and more specifically, to a data storage scheduling management system, an application method and a device thereof. Background Art

[0002] Currently, the data storage modes for application systems mainly include:

[0003] 1. Single-database and single-table storage mode: In the initial stage of system construction, when the data volume is small, a single database and a single table are used for data storage. For example, for the order table t_order, only operations such as adding, deleting, modifying, and querying on t_order are required at this stage.

[0004] 2. Single-database horizontal sharding storage mode: As the data volume increases and the data volume of a single table reaches the storage threshold, to relieve the database pressure and prevent database performance bottlenecks, it is necessary to split the single table. The original single table is split into multiple shard tables, that is, horizontal table partitioning within a single database. For example, for the order table t_order, assuming the single-table storage threshold is 30 million data volumes, when the data volume of this table reaches 32 million, exceeding the single-table storage threshold of 30 million, at this time, it is necessary to split the original order table, such as splitting it into 32 tables, and the original t_order is split into t_order1, t_order2, t_order3... t_order32. Then, 32 million data is sharded to 32 tables according to the strategy, and the data volume of each table is theoretically 1 million, so as to reduce the data volume of a single table.

[0005] 3. Vertical database partitioning storage mode: As the data volume further increases and the single-database reaches the threshold, to relieve the database pressure, it is necessary to split the database, that is, vertical database partitioning. For example, in the database ds1, there are 32 order tables t_order1~32 and 32 transaction tables t_record1~32, and the total data volume has reached the database threshold. At this time, a new database ds2 needs to be created, and the data of 32 transaction tables in the original database ds1 is migrated to the new database ds2. After the migration is completed, the data is stored as 32 order tables ds1.t_order1~32 in the ds1 database and 32 transaction tables ds2.t_record1~32 in the ds2 database, thus reducing the data volume of a single database.

[0006] As the amount of data continues to increase during the construction cycle of application systems, the data storage mode usually goes through three stages. However, there are several core pain points in the change of the data storage mode: different database connection pool technologies are adopted in each data storage mode, and there are differences in function positioning, data source configuration, SQL parsing, and routing among different connection pool technologies. When the storage mode changes, the workload of development, transformation, and regression testing for the application system to adapt to the database connection pool is very large, and the investment cost is high; when changing from the single-database single-table storage mode to the single-database horizontal sharding storage mode, or from the single-database horizontal sharding storage mode to the vertical database sharding storage mode, data preparation and data migration are involved. During the data preparation and data migration stages, the application system needs to be shut down for a long time, and it is impossible to achieve a smooth upgrade; a large number of routing settings are required for the single-database horizontal sharding storage mode and the vertical database sharding storage mode, the configuration is cumbersome, and the technical threshold is relatively high.

[0007] During the construction process of most current application systems, the data storage usually goes through the single-database single-table storage mode, the single-database horizontal sharding storage mode, and the vertical database sharding storage mode. Among them, the Druid connection pool is adopted in the single-database single-table storage mode stage; the ShardingJdbc connection pool is adopted in the single-database horizontal sharding storage mode; the Mycat or ShardingSphere connection pool is adopted in the vertical database sharding storage mode stage; the MybatisPlus framework is uniformly used for the DB persistence layer.

[0008] The problems encountered during the above storage mode switching are as follows: 1. The differences in the database pool technologies of each data storage mode lead to a huge project for system adaptation, transformation, and regression testing; 2. For horizontal sharding or vertical database sharding, a large number of parameter configurations are required for the routing mapping of database tables, and the technical threshold is relatively high, resulting in a relatively high technical training cost for developers. 3. When changing from the single-database single-table storage mode to the horizontal sharding storage mode, the original single-table data needs to be routed to the sharded table; when changing from the horizontal sharding storage mode to the vertical database sharding storage mode, the original database data needs to be migrated to the newly created database. In traditional technical solutions, database switching, data migration, etc. are involved, and the application system needs to be shut down. After the database table operations are completed, the application system is restarted. The larger the amount of data, the longer the shutdown time, which is unacceptable for 7*24-hour application systems. Summary of the Invention

[0009] The purpose of the present invention is to provide a data storage scheduling and management system, which enables the application system and the database to continue running without interruption during the data storage mode switching, making it more convenient for actual applications.

[0010] The above technical objective of the present invention is achieved through the following technical solutions: A data storage scheduling and management system, comprising:

[0011] A scheduling manager for logically managing an annotation parser, a configuration registrar, a data source manager, a dynamic routing manager, and a data sharding collector;

[0012] An annotation parser for scanning annotations marked on a logical table, determining a data storage mode, and building a data storage mapping cache for the logical table;

[0013] A configuration registrar for building parameter configurations for a horizontal sharding storage mode and a vertical database sharding storage mode;

[0014] A data source manager for creating connection pool components corresponding to each data storage mode;

[0015] A dynamic routing manager for registering core information of data storage mapping, parameter configuration, and connection pool components, and also for determining and routing data storage modes during the operation of the application system;

[0016] A data sharding collector for horizontally sharding data in a single database and single table storage mode.

[0017] As a preferred technical solution of the present invention, the annotation parser, the configuration registrar, and the data source manager all work during the startup phase of the application system.

[0018] As a preferred technical solution of the present invention, when the annotation parser determines that the data storage mode of the logical table is a horizontal sharding storage mode, it calls the configuration registrar to perform parameter configuration for sharding;

[0019] When the data storage mode of the logical table of the annotation parser is a vertical database sharding storage mode, it calls the configuration registrar to perform parameter configuration for database sharding and sharding.

[0020] As a preferred technical solution of the present invention, when the data storage scheduling management system receives an operation instruction for a logical table after the startup of the application system is completed, it is responded to by the dynamic routing manager. According to the data storage mapping, the data storage mode of the logical table is confirmed, the connection pool component is locked, and corresponding data operations are performed according to the parameter configuration.

[0021] As a preferred technical solution of the present invention, the data storage scheduling management system is also used for the application system to switch from a single database and single table storage mode to a horizontal sharding storage mode. The switching method is as follows:

[0022] In response to a horizontal sharding operation instruction, the scheduling manager obtains a sharded table and registration information;

[0023] Through the data sharding collector, according to the sharded table and registration information, the data in the original table is asynchronously stored into the sharded table;

[0024] After the sharding is completed, obtain the latest data storage mapping cache through the annotation parser;

[0025] Update the latest data storage mapping cache to the dynamic routing manager.

[0026] As a preferred technical solution of the present invention, the data storage scheduling management system is further used for the application system to switch from the horizontal sharding storage mode to the vertical database sharding storage mode, and the switching method is as follows:

[0027] In response to the vertical database sharding operation instruction, confirm whether the sharded table in the original database has been migrated to the target database through the scheduling manager;

[0028] After the migration is completed, obtain the latest data storage mapping cache through the annotation parser;

[0029] Update the latest data storage mapping cache to the dynamic routing manager.

[0030] An application method of a data storage scheduling management system includes the following steps:

[0031] In the startup stage of the application system, scan the annotations marked on the logical table through the annotation parser to determine the data storage mode, and at the same time construct the data storage mapping of the logical table;

[0032] Through the configuration registrar, perform corresponding sharding parameter configuration or database sharding and sharding parameter configuration on the database according to the data storage mode;

[0033] After the parameter configuration is completed, create a connection pool component corresponding to each data storage mode through the data source manager;

[0034] Register the core information of the data storage mapping, parameter configuration, and connection components to the dynamic routing manager;

[0035] In the running stage of the application system, perform data storage mode judgment and routing through the dynamic routing manager;

[0036] If the data storage mode is the single database and single table storage mode, perform horizontal sharding of the data through the data sharding collector.

[0037] A data storage scheduling management device includes: a processor and a memory, the memory stores a computer program executable by the processor, and when the processor executes the computer program, the above method is implemented.

[0038] In summary, the present invention has the following beneficial effects: 1. During the process of switching the data storage of the application system, there is no code, no configuration, and no testing in the entire link; 2. The framework is lightweight, has no intrusion into the business code, and has strong scalability; 3. It is friendly to developers' APIs and has no technical barriers. 4. When switching the data storage mode, the application system and the database do not need to be shut down, which is more convenient for actual applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the component loading stage of the present invention;

[0040] Figure 2 It is a schematic diagram of the component calling stage of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The present invention will be further described in detail below with reference to the accompanying drawings.

[0042] As Figure 1 and Figure 2 shown, the present invention provides a data storage scheduling and management system, including: a scheduling manager, an annotation parser, a configuration registrar, a data source manager, a dynamic routing manager, and a data sharding collector;

[0043] The scheduling manager is used to logically manage the annotation parser, the configuration registrar, the data source manager, the dynamic routing manager, and the data sharding collector during the application system determination stage, and is responsible for the overall scheduling of the components;

[0044] The annotation parser is used to scan the annotations marked on the logical table during the application system startup stage, determine the data storage mode, and simultaneously construct the data storage mapping cache of the logical table;

[0045] The configuration registrar is used to construct the parameter configurations of the horizontal sharding storage mode and the vertical database sharding storage mode during the application system startup stage; specifically, when the annotation parser determines that the data storage mode of the logical table is the horizontal sharding storage mode, the configuration registrar is called to perform the parameter configuration of the sharding; when the data storage mode of the logical table of the annotation parser is the vertical database sharding storage mode, the configuration registrar is called to perform the parameter configuration of the database sharding and the sharding.

[0046] The data source manager is used to create a connection pool component corresponding to each data storage mode during the application system startup stage;

[0047] The dynamic routing manager is used to register the core information of the data storage mapping, the parameter configuration, and the connection pool component, and is also used to judge and route the data storage mode during the operation of the application system;

[0048] A data sharding collector is used to perform horizontal sharding on the data of a single table in the single-database and single-table storage mode during the operation phase of the application system.

[0049] Specifically, the annotation parser, the configuration registrar, and the data source manager all work during the startup phase of the application system.

[0050] In the data storage scheduling and management system, after the application system starts up and receives an operation instruction for a logical table, the dynamic routing manager responds, confirms the data storage mode of the logical table according to the data storage mapping, locks the connection pool component, and simultaneously performs corresponding data operations according to the parameter configuration.

[0051] Corresponding to the above system, the present invention also provides an application method for a data storage scheduling and management system, including a component loading phase and a component calling phase, and having the following steps in total:

[0052] In the component loading phase, as Figure 1 shown:

[0053] S1. During the startup phase of the application system, first, the scheduling manager is created, and then the annotation parser is scheduled through the scheduling manager. Through the annotation parser, the annotations marked on the logical table are scanned to determine the data storage mode, and at the same time, the data storage mapping of the logical table is constructed;

[0054] S2. Then, the configuration registrar is called through the scheduling manager. Through the configuration registrar, corresponding sharding parameter configurations or database sharding and sharding parameter configurations are performed on the database according to the data storage mode;

[0055] Specifically, when the annotation parser determines that the data storage mode of the logical table is the horizontal sharding storage mode, the configuration registrar is called to perform sharding parameter configuration; when the data storage mode of the logical table by the annotation parser is the vertical database sharding storage mode, the configuration registrar is called to perform database sharding and sharding parameter configuration.

[0056] S3. After the parameter configuration is completed, the data source manager is called through the scheduling manager. Through the data source manager, connection pool components corresponding to each data storage mode are created;

[0057] S4. Register the data storage mapping, parameter configuration, and core information of the connection components to the dynamic routing manager;

[0058] The embodiments of the above S1-S4 are as follows:

[0059] In the application system, the order table is t_order, and the corresponding order operation interface class is TOrderMapper, which is the data operation interface class. Developers can mark annotations on the interface class. Among them, @ShardsMapper(mod=single) represents that the data storage mode of the order table is the single database and single table storage mode; @ShardsMapper(mod=sharding, ds=ds1) represents the horizontal sharding storage mode of the order database, and @ShardsMapper(mod=vertical, ds=ds2) represents the vertical database sharding storage mode.

[0060] During the startup phase of the application system, after the scheduling manager is created, it will call the annotation parser to scan the interface classes marked with the @ShardsMapper annotation in the program, confirm the data storage mode according to the mod type, and at the same time build the data storage mapping cache of the logical table for subsequent quick retrieval. The key value of the cache information is t_order, and the value is the specific data storage mode meta information.

[0061] After the mapping is built, the scheduling manager schedules the configuration register to complete the parameter configuration. The parameter information is that the data source corresponding to the t_order table is ds=ds1, etc.

[0062] After the parameter configuration is completed, the scheduling manager will call the data source manager to create connection pools in the three data storage modes of ds1 and ds2 according to the data source.

[0063] Finally, the scheduling manager will register the core information of the loaded data storage mapping, parameter configuration, and connection pool components to the dynamic routing manager for subsequent processing after the application starts.

[0064] During the component call phase, as Figure 2 shown:

[0065] S5. During the running phase of the application system, that is, after the application system starts up, the data storage mode is judged and routed through the dynamic routing manager. That is, when there is an external instruction to request an operation on the logical table, the dynamic routing manager confirms the data storage mode of the logical table according to the data storage mapping, locks the connection pool component, and at the same time performs corresponding data operations according to the parameter configuration.

[0066] If the data storage mode is the single database and single table storage mode, horizontal sharding of the data is performed through the data sharding collector.

[0067] As an embodiment of S5, when an external instruction is received and a single order query needs to be executed, the application system will call the query of the order operation interface class TOrderMapper. At this time, the dynamic routing manager will intercept the request and confirm the single operation interface class TOrderMapper. The dynamic routing manager will first obtain the data storage mapping, parameter configuration, connection pool components and other core information saved in the cache, and obtain the data storage mapping of the cache storage to confirm that the operation table is t_order and the data storage mode. If it is a single database and single table storage mode, the dynamic routing manager will directly call the connection pool component to execute the query on the t_order single table in the database ds1; if it is a single database sharding query, the dynamic routing manager will execute the query on the t_order1~32 sharding tables in the database ds1 through parameter configuration and the connection pool component; if it is a vertical database sharding storage mode, the dynamic routing manager will execute the query on the t_order1~32 sharding tables in the database ds2 through parameter configuration and the connection pool component.

[0068] As an embodiment of the present invention, the data storage scheduling management system is also used for the application system to switch from the single database and single table storage mode to the horizontal sharding storage mode. The switching method is as follows:

[0069] Sa1. In response to the horizontal sharding operation instruction, the scheduling manager obtains the sharding table and registration information;

[0070] Sa2. Through the data sharding collector, according to the sharding table and registration information, asynchronously store the data in the original table into the sharding table;

[0071] Sa3. After the sharding is completed, obtain the latest data storage mapping cache through the annotation parser;

[0072] Sa4. Update the latest data storage mapping cache to the dynamic routing manager.

[0073] Specifically, the above switching process mainly occurs after the application system starts up. When switching from the horizontal sharding storage mode to the vertical database sharding storage mode, technicians can create corresponding sharding tables in the ds1 database. After the creation is completed, the technicians log in to the management system and register the data storage mode change information. The registered data storage mode change information mainly includes the sharding table name, sharding key, etc.; after the registration is completed, the technicians click to execute the sharding operation. After the application system receives the sharding operation instruction, the scheduling manager obtains the sharding table and the registered data storage mode change information, and then calls the data sharding collector. According to the registration information and the sharding key, asynchronously store the data in the original table into the sharding table.

[0074] After the sharding task is completed, the technician adjusts the cache information of (a - data storage mapping) in the management system, changes the original single - database and single - table storage mode to the horizontal sharding storage mode, and clicks to take effect. After the application system receives the instruction, it obtains the latest data storage mapping cache through the annotation parser, and changes the cache information of the data storage mapping in the dynamic routing manager. Thus, the switching work is completed, and the whole process does not require system downtime.

[0075] For example, for data sharding of the original single - database and single - table order table t_order, assuming that the current data volume of t_order is 32 million, and using the customer number as the sharding key, the single table is split into 32 sharded tables. The technician creates sharded tables for orders, t_order1, t_order2, t_order3... t_order32, in the ds1 database offline; then registers the data storage mode change information in the management system, mainly entering the change type, single - table name t_order, sharded - table names t_order1 to 32, sharding key customer number, etc. After registration, click to execute the sharding operation, that is, issue a horizontal sharding operation instruction. The data sharding collector will perform iterative processing on 32 million order data according to the registered information, obtain the customer number of each order, calculate the sharding index by taking the hash modulo of the customer number. Assuming the customer number is CNO1000001 and the calculated index by taking the hash modulo is 1, the data sharding collector will add this data to the t_order1 sharded table, and so on until all order data sharding is completed. Thus, the data migration from a single table to horizontal sharding is completed. After the data migration is completed, the technician adjusts the cache information of the data storage mode in the back - office system, changes the original single - table storage to sharded storage, and clicks to take effect to update the cache of the data storage mapping. Thus, the data storage mode switching is completed. When a new request accesses the order table, the dynamic routing manager will process it corresponding to the sharded tables t_order1 to 32 according to the adjusted cache of the data storage mapping.

[0076] As an embodiment of the present invention, the data storage scheduling and management system is also used for the application system to switch from the horizontal sharding storage mode to the vertical database sharding storage mode. The switching method is as follows:

[0077] Sb1. In response to the vertical database sharding operation instruction, confirm through the scheduling manager whether the sharded tables in the original database have been migrated to the target database;

[0078] Sb2. After the migration is completed, obtain the latest data storage mapping cache through the annotation parser;

[0079] Sb3. Update the latest data storage mapping cache to the dynamic routing manager.

[0080] Specifically, the above switching process mainly occurs after the application system starts up. When switching from the single-database horizontal sharding storage mode to the vertical database sharding storage mode, technicians offline migrate the sharded tables in the original database ds1 to the target database ds2. After the migration is completed, technicians adjust the cache information of the data storage mapping in the management system, change the original single-database horizontal sharding storage mode to the vertical database sharding storage mode, and click to take effect, that is, issue a vertical database sharding operation instruction. After the application system receives the instruction, it confirms whether the sharded tables in the original database have been migrated to the target database through the scheduling manager. After confirmation, it will modify the cache information change of the data storage mapping in the dynamic routing manager. Thus, the switching work is completed, and the whole process does not require service interruption.

[0081] For example, when migrating the sharded tables t_order1, t_order2, t_order3...t_order32 in the original database, technicians need to migrate the order sharded tables in the original database ds1 to the target database ds2. After the data migration is completed, technicians adjust the cache information of the data storage mode in the back-office system, change the original horizontal sharding storage mode to the vertical database sharding storage mode, and click to take effect to update the cache of the data storage mapping. Thus, the storage mode switching is completed. When a new request accesses the order table, the dynamic routing manager will process the sharded tables t_order1~32 in the target database ds2 according to the adjusted cache of the data storage mapping.

[0082] Corresponding to the above system and application method, the present invention also provides a data storage scheduling management device, including: a processor and a memory. The memory stores a computer program executable by the processor, and when the processor executes the computer program, the above method is implemented.

[0083] The advantages of the present invention are as follows: 1. During the data storage switching process of the application system, there is 0 code, 0 configuration, and 0 testing in the whole link; 2. The framework is lightweight, has no intrusion into the business code, and has strong scalability; 3. It is friendly to developers' APIs and has no technical barriers. 4. When switching the data storage mode, the application system and the database do not need to stop service.

[0084] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements and refinements made without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A data storage scheduling management system, characterized in that: include: The scheduling manager is used to perform logical management on the annotation parser, configuration registrar, data source manager, dynamic routing manager, and data sharding collector; Annotation parser, used to scan the annotations marked by the logical table, determine the data storage mode, and build the data storage mapping cache of the logical table; Configuration register, used to build parameter configuration for horizontal sharding storage mode and vertical sharding storage mode; Data source manager, used to create connection pool components corresponding to each data storage mode; Dynamic routing manager, used to register data storage mapping, parameter configuration and core information of connection pool components, and also used to determine and route data storage modes when the application system is running; The data sharding collector is used to horizontally shard data in the single-database single-table storage mode.

2. A data storage scheduling management system according to claim 1, characterized in that: The annotation parser, the configuration register and the data source manager all work during the application system startup phase.

3. A data storage scheduling management system according to claim 2, characterized in that: When the annotation parser determines that the data storage mode of the logic table is the horizontal sharding storage mode, the configuration register is called to perform sharding parameter configuration; When the data storage mode of the annotation parser logic table is the vertical sub-library storage mode, the configuration register is called to perform parameter configuration of sub-library and sharding.

4. A data storage scheduling management system according to claim 3, characterized in that: After the application system is started, when the data storage scheduling and management system receives an operation instruction for the logic table, the dynamic routing manager responds, confirms the data storage mode of the logic table according to the data storage mapping, locks the connection pool component, and performs corresponding data operations according to the parameter configuration.

5. A data storage scheduling management system according to claim 4, characterized in that: The data storage scheduling management system is also used to switch the application system from a single database and single table storage mode to a horizontal sharding storage mode, and the switching method is: In response to the horizontal sharding operation instruction, the scheduling manager obtains the sharding table and registration information; By means of the data sharding collector, data in the original table is asynchronously stored in the sharding table according to the sharding table and registration information; After the sharding is completed, the latest data storage mapping cache is obtained through the annotation parser; The dynamic routing manager is updated with the latest data storage mapping cache.

6. A data storage scheduling management system according to claim 5, characterized in that: The data storage scheduling management system is also used to switch the application system from the horizontal sharding storage mode to the vertical sharding storage mode, and the switching method is: In response to the vertical database sharding operation instruction, confirming through the scheduling manager whether the sharding table in the original database has been migrated to the target database; After the migration is completed, the latest data storage mapping cache is obtained through the annotation parser; The dynamic routing manager is updated with the latest data storage mapping cache.

7. An application method of a data storage scheduling management system, characterized in that: The steps include: During the application system startup phase, the annotation parser scans the annotations marked in the logic table to determine the data storage mode and build the data storage mapping of the logic table. By configuring the registrar, according to the data storage mode, the corresponding sharding parameters or database and sharding parameters are configured for the database; After the parameter configuration is completed, create the connection pool component corresponding to each data storage mode through the data source manager; Register the data storage mapping, parameter configuration, and core information of the connection components to the dynamic routing manager; During the application system operation phase, the data storage mode is determined and routed through the dynamic routing manager; If the data storage mode is single database single table storage mode, the data is horizontally sharded through the data sharding collector.

8. A data storage scheduling management device, characterized in that: include: A processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method according to any one of claim 7 is implemented.

Images