Method and system for dynamically switching data sources in combination with task scheduling platform

Abstract

The invention discloses a dynamic data source switching method and system combined with a task scheduling platform, and relates to the technical field of data management. Comprising the steps that 1, a data source is managed, 2, task scheduling and management are conducted based on a task scheduling platform, task information is stored in a database, tenant codes where tasks are located are marked in the task information, the tenant codes are the same as data source codes, and when an application service instance is started, timed tasks are registered to the task scheduling platform; and 3, dynamically switching the data source configuration information according to the parameters of the timed task in the task information.

Description

Technical Field

[0001] The present invention discloses a method and system for dynamically switching data sources in combination with a task scheduling platform, which relates to the technical field of data management. Background Art

[0002] The rise and wide application of the multi-tenant architecture have further highlighted the indispensable importance of multi-data sources in enterprise-level applications. With the popularization of cloud computing and the SaaS model, multi-tenancy has become a mainstream software deployment method, which allows service providers to run multiple independent application instances on the same hardware and software platform, and each instance serves different tenants. In this architecture, each tenant has its own independent data source, and the data between each data source is isolated from each other. Such an architecture design not only makes tenant data more secure, but also brings complexity to data processing and management.

[0003] In the multi-tenant architecture, scheduled tasks are an important tool for automating business logic processing. However, for the same business logic between tenants, generally a set of task calculation logics are used. At present, during the execution of this same calculation logic by scheduled tasks, the dynamic switching of multiple tenant data sources cannot be perfectly performed. Summary of the Invention

[0004] Aiming at the problems of the prior art, the present invention provides a method and system for dynamically switching data sources in combination with a task scheduling platform. Before the execution of a scheduled task, according to the parameters transmitted by the task scheduling platform, the present invention dynamically selects and connects to the corresponding data source, so as to achieve the purpose of dynamically switching data sources for system scheduled tasks. The method of the present invention simplifies the configuration and management process of scheduled tasks, improves the flexibility and scalability of tasks, and enables them to better adapt to the complex and changeable data source environment in the multi-tenant architecture.

[0005] The specific solution proposed by the present invention is as follows:

[0006] The present invention provides a method for dynamically switching data sources in combination with a task scheduling platform, including:

[0007] Step 1: Manage data sources:

[0008] Establish a data source information library, and use the data source information library to store the configuration information of data sources,

[0009] and initialize the data source when the application service instance starts;

[0010] Step 2: Perform task scheduling and management based on the task scheduling platform, store the task information in the database, and indicate the tenant code where the task is located in the task information. The tenant code is the same as the data source code.

[0011] When the application service instance starts, register a scheduled task with the task scheduling platform;

[0012] Step 3: Dynamically switch the data source configuration information according to the parameters of the scheduled task in the task information:

[0013] Configure an interceptor for the task scheduling platform to intercept all scheduled task execution requests of the task scheduling platform,

[0014] According to the scheduled task execution request, parse the task configuration information, obtain the data source code, and query the corresponding data source configuration information in the data source information library according to the data source code,

[0015] Establish a connection with the target data source using the data source configuration information and execute the corresponding scheduled task. After the scheduled task is completed, clear the data source information for subsequent operations to use different data sources.

[0016] Furthermore, in step 1 of the method for dynamically switching data sources in combination with a task scheduling platform, a new data source is registered using the data source information library, and operations such as modifying or deleting the configured information of the registered data source are performed. At the same time, a data source query interface is provided for querying the required data source configuration information before task execution.

[0017] And introduce a template mechanism to create a data source configuration template. Define common connection parameters and advanced settings through the data source configuration template for scenarios where data sources need to be quickly deployed.

[0018] Furthermore, initializing the data source in step 1 of the method for dynamically switching data sources in combination with a task scheduling platform includes:

[0019] Obtain all data configuration information,

[0020] Traverse the data source configuration information. If it is empty, report an error; if it is not empty, generate a DataSource object according to the database connection string and connection pool parameter information in the data source configuration information.

[0021] After initialization, store the DataSource objects of all data sources in the utility class DynamicDataSourceContextHolder for dynamically switching data sources.

[0022] Furthermore, registering a scheduled task with the task scheduling platform in step 2 of the method for dynamically switching data sources in combination with a task scheduling platform includes:

[0023] Traverse all task lists to determine whether the tenant code where the task is located exists.

[0024] If it exists, assemble the tenant code and the scheduling configuration item into the interface parameters for calling the task scheduling platform, and create a timed task. The scheduling configuration item includes the scheduling type, executor ID, routing policy, scheduling expiration policy, and blocking handling policy.

[0025] If it does not exist, or the tenant code where it is located is "all", it means that the timed task needs to be executed by all tenants. Then, obtain all data source codes, and traverse the data source codes and the scheduling configuration items to assemble them into the interface parameters for calling the task scheduling platform, and create multiple timed tasks.

[0026] The present invention also provides a dynamic data source switching system combined with a task scheduling platform, including a data source management module, a task management module, and a task execution management module.

[0027] The data source management module manages data sources:

[0028] Establish a data source information library, and use the data source information library to store the configuration information of the data source.

[0029] And when the application service instance starts, initialize the data source.

[0030] The task management module performs task scheduling and management based on the task scheduling platform, stores the task information in the database, and indicates the tenant code where the task is located in the task information. The tenant code is the same as the data source code.

[0031] When the application service instance starts, register the timed task with the task scheduling platform.

[0032] The task execution management module dynamically switches the data source configuration information according to the parameters of the timed task in the task information:

[0033] Configure the interceptor of the task scheduling platform, and intercept all timed task execution requests of the task scheduling platform through the interceptor.

[0034] According to the timed task execution request, parse the task configuration information, obtain the data source code, and query the corresponding data source configuration information in the data source information library according to the data source code.

[0035] Use the data source configuration information to establish a connection with the target data source, and execute the corresponding timed task. After the timed task is completed, clear the data source information so that different data sources can be used for subsequent operations.

[0036] Furthermore, the data source management module of the dynamic data source switching system combined with the task scheduling platform also registers new data sources using the data source information library, modifies or deletes the configuration information of the registered data sources, and provides a data source query interface for querying the required data source configuration information before task execution.

[0037] And introduce a template mechanism to create a data source configuration template. Define common connection parameters and advanced settings through the data source configuration template for scenarios where data sources need to be quickly deployed.

[0038] Furthermore, the data source management module of the dynamic data source switching system combined with the task scheduling platform initializes the data source, including:

[0039] Obtain all data configuration information.

[0040] Traverse the data source configuration information. If it is empty, report an error; if it is not empty, generate a DataSource object according to the database connection string and connection pool parameter information in the data source configuration information.

[0041] After initialization is completed, store the DataSource objects of all data sources in the utility class DynamicDataSourceContextHolder for dynamic data source switching.

[0042] Furthermore, the task management module of the dynamic data source switching system combined with the task scheduling platform registers a scheduled task with the task scheduling platform, including:

[0043] Traverse all task lists and determine whether the tenant code where the task is located exists.

[0044] If it exists, assemble the tenant code and scheduling configuration items into the interface parameters for calling the task scheduling platform to create a scheduled task. The scheduling configuration items include scheduling type, executor id, routing policy, scheduling expiration policy, and blocking handling policy.

[0045] If it does not exist, or the tenant code where it is located is "all", it means that the scheduled task needs to be executed by all tenants. Then obtain all data source codes, traverse the data source codes and scheduling configuration items to assemble into the interface parameters for calling the task scheduling platform to create multiple scheduled tasks.

[0046] The advantages of the present invention are:

[0047] Simplify the registration, maintenance, and query processes of data sources, while reducing the system maintenance costs caused by data source changes. Through the dynamic parameter passing function of the task scheduling platform, the timed tasks can quickly switch data sources during execution without modifying the code or configuration, enhancing the scalability and stability of the system. In addition, the dynamic data source switching ensures high efficiency and fine-grained control of task execution, improving the security and isolation of tenant data. Overall, this method simplifies the configuration and management processes of timed tasks, improves the flexibility and scalability of tasks, enabling them to better adapt to the complex and changing data source environment under the multi-tenant architecture. Brief Description of the Drawings

[0048] Figure 1 It is a schematic diagram of the application architecture of the system of the present invention.

[0049] Figure 2 It is a schematic diagram of the data source initialization process.

[0050] Figure 3 It is a schematic diagram for the process of dynamically switching data sources. Detailed Embodiments

[0051] A data source refers to the data source required for the system to run, generally referring to a database.

[0052] Multiple data sources mean that the system depends on multiple data sources during operation.

[0053] Dynamic switching means that during the operation of the system, through a certain instruction, different data sources can be automatically switched without restarting the service.

[0054] The present invention will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited do not limit the present invention.

[0055] Embodiment 1

[0056] The present invention provides a method for dynamically switching data sources in combination with a task scheduling platform, including:

[0057] Step 1: Manage data sources:

[0058] Establish a data source information library, use the data source information library to save the configuration information of the data source, such as the database URL, username, password, etc. Among them, a new data source is also registered using the data source information library, and operations such as modifying or deleting the configuration information of the registered data source are performed. At the same time, a data source query interface is provided for querying the required data source configuration information before task execution.

[0059] And introduce a template mechanism to create a data source configuration template. Define common connection parameters and advanced settings through the data source configuration template for scenarios where data sources need to be quickly deployed.

[0060] When the application service instance starts, initialize the data source, which may specifically include:

[0061] Obtain all data configuration information.

[0062] Traverse the data source configuration information. If it is empty, report an error; if it is not empty, generate a DataSource object based on the database connection string and connection pool parameter information in the data source configuration information. For example, use an open-source multi-data source library such as dynamic-datasource-spring-boot-starter to implement the generation of the DataSource object, or use the @Configuration annotation to define multiple DataSource Beans and create the implementation of the DataSource object through the @Bean annotation.

[0063] After initialization, store the DataSource objects of all data sources in the utility class DynamicDataSourceContextHolder for dynamically switching data sources. ThreadLocal can be used to store the name of the data source used by the current thread. The ThreadLocal variable internally maintains a stack (Deque) or list to support the switching of data sources.

[0064] Step 2: Perform task scheduling and management based on the task scheduling platform, store the task information in the database, and indicate the tenant code where the task is located in the task information. The tenant code is the same as the data source code.

[0065] When the application service instance starts, register a scheduled task with the task scheduling platform.

[0066] Among them, the task scheduling platform is responsible for the task scheduling and management of the entire system. Tasks can be configured at the front end and through API interfaces, and the task information is stored in the database, and the execution of the task is triggered according to the preset scheduling rules; at the same time, the execution status of the task can be monitored, and the execution result is recorded after the task execution is completed; finally, the task scheduling platform supports passing parameters to each task to more conveniently control the execution of the task.

[0067] When a multi-tenant system executes scheduled tasks, there may be two situations: the task needs to be executed in all tenants, or the task is executed in a single tenant or some tenants, and other tasks are not executed. Therefore, the tenant code where the task is located is indicated in the task information, and this tenant code is the same as the data source code. When the service instance starts, it is necessary to register the scheduled task with the task scheduling platform, which may include:

[0068] Traverse all task lists to determine whether the tenant code where the task is located exists.

[0069] If it exists, assemble the tenant code and the scheduling configuration items into the interface parameters for calling the task scheduling platform to create a scheduled task. The scheduling configuration items include scheduling type, executor id, routing policy, scheduling expiration policy, and blocking handling policy.

[0070] If it does not exist, or the tenant code where it is located is "all", it means that the scheduled task needs to be executed in all tenants. Then, obtain all data source codes, traverse the data source codes and the scheduling configuration items to assemble them into the interface parameters for calling the task scheduling platform, and create multiple scheduled tasks.

[0071] Step 3: Dynamically switch the data source configuration information according to the parameters of the scheduled task in the task information:

[0072] Configure the interceptor of the task scheduling platform to intercept all scheduled task execution requests of the task scheduling platform.

[0073] According to the scheduled task execution request, parse the task configuration information to obtain the data source code, query the corresponding data source configuration information in the data source information library according to the data source code. If the data source code is not included, the default primary data source is used to execute the scheduled task.

[0074] The data source code can be further verified:

[0075] According to the parameter code, query the data source management interface. If the information can be obtained correctly, the data source code configuration is correct. If the information cannot be obtained, it proves that the data source code configuration is incorrect or the data source has expired, and an error message needs to be sent to the task configurator for notification.

[0076] After obtaining the data source code information according to the task parameters, taking dynamic-datasource-spring-boot-starter as an example, use the method com.baomidou.dynamic.datasource.toolkit.DynamicDataSourceContextHolder.push() to put the code value into the context of dynamic data switching. At this time, the data source router, such as Spring's AbstractRoutingDataSource, selects the corresponding data source according to the data source name stored in DynamicDataSourceContextHolder, establishes a connection with the target data source using the data source configuration information in the current context, and executes the corresponding scheduled task. After the scheduled task is completed, call DynamicDataSourceContextHolder.clear() to clear the data source information for subsequent operations to use different data sources.

[0077] Embodiment 2

[0078] The present invention also provides a dynamic data source switching system combined with a task scheduling platform, including a data source management module, a task management module, and a task execution management module.

[0079] The data source management module manages data sources:

[0080] Establish a data source information library, and use the data source information library to store the configuration information of the data source.

[0081] And when the application service instance starts, initialize the data source.

[0082] The task management module performs task scheduling and management based on the task scheduling platform, stores the task information in the database, and indicates the tenant code where the task is located in the task information. The tenant code is the same as the data source code.

[0083] When the application service instance starts, register the scheduled task with the task scheduling platform.

[0084] The task execution management module dynamically switches the data source configuration information according to the parameters of the scheduled task in the task information:

[0085] Configure the interceptor of the task scheduling platform, and intercept all scheduled task execution requests of the task scheduling platform through the interceptor.

[0086] According to the scheduled task execution request, parse the task configuration information, obtain the data source code, and query the corresponding data source configuration information in the data source information library according to the data source code.

[0087] Establish a connection with the target data source using the data source configuration information, and execute the corresponding scheduled tasks. After the scheduled tasks are completed, clear the data source information for subsequent operations to use different data sources.

[0088] Regarding the information interaction and execution process among the various modules in the above system, since they are based on the same concept as the method embodiments of the present invention, the specific content can be referred to the description in the method embodiments of the present invention and will not be elaborated here.

[0089] Similarly, the system of the present invention simplifies the registration, maintenance, and query processes of data sources, while reducing the system maintenance costs caused by data source changes. Through the dynamic parameter passing function of the task scheduling platform, the scheduled tasks can quickly switch data sources during execution without modifying the code or configuration, enhancing the scalability and stability of the system. In addition, the dynamic data source switching ensures high efficiency and fine-grained control of task execution, improving the security and isolation of tenant data. Generally speaking, this method simplifies the configuration and management process of scheduled tasks, improves the flexibility and scalability of tasks, and enables it to better adapt to the complex and changing data source environment under the multi-tenant architecture.

[0090] It should be noted that not all steps and modules in the above processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structures described in the above embodiments can be physical structures or logical structures, that is, some modules may be implemented by the same physical entity, or some modules may be implemented separately by multiple physical entities, or some components in multiple independent devices may be jointly implemented.

[0091] The above-described embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for dynamically switching data sources in combination with a task scheduling platform, characterized in that Including: Step 1: Manage data sources: Establish a data source information repository, and use the data source information repository to store the configuration information of the data sources, and initialize the data sources when the application service instance starts; Step 2: Perform task scheduling and management based on the task scheduling platform, store the task information in the database, and indicate the tenant code where the task is located in the task information. The tenant code is the same as the data source code, and register a scheduled task with the task scheduling platform when the application service instance starts; Step 3: Dynamically switch the data source configuration information according to the parameters of the scheduled task in the task information: Configure an interceptor for the task scheduling platform, and intercept all scheduled task execution requests of the task scheduling platform through the interceptor, parse the task configuration information according to the scheduled task execution request, obtain the data source code, and query the corresponding data source configuration information in the data source information repository according to the data source code, establish a connection with the target data source using the data source configuration information, and execute the corresponding scheduled task. After the scheduled task is completed, clear the data source information so that different data sources can be used for subsequent operations.

2. The method for dynamically switching data sources in combination with a task scheduling platform according to claim 1, characterized in that In Step 1, a new data source is also registered using the data source information repository, and the configuration information of the registered data sources can be modified or deleted. At the same time, a data source query interface is provided to query the required data source configuration information before task execution, and a template mechanism is introduced to create a data source configuration template. The common connection parameters and advanced settings are defined through the data source configuration template for scenarios where data sources need to be quickly deployed.

3. The method for dynamically switching data sources in combination with a task scheduling platform according to claim 1, characterized in that Initializing the data sources in Step 1 includes: Obtain all data configuration information, traverse the data source configuration information. If it is empty, report an error; if it is not empty, generate a DataSource object according to the database connection string and connection pool parameter information in the data source configuration information, after initialization is completed, store the DataSource objects of all data sources in the utility class DynamicDataSourceContextHolder for dynamically switching data sources.

4. A method for dynamically switching data sources in combination with a task scheduling platform according to claim 1, characterized in that Registering a scheduled task with the task scheduling platform in Step 2 includes: Traverse all task lists and determine whether the tenant code where the task is located exists, if it exists, assemble the tenant code and the scheduling configuration items into the interface parameters for calling the task scheduling platform, and create a scheduled task. The scheduling configuration items include the scheduling type, executor id, routing strategy, scheduling expiration strategy, and blocking handling strategy, if it does not exist, or the tenant code where it is located is all, it means that the scheduled task needs to be executed by all tenants, then obtain all data source codes, traverse the data source codes and the scheduling configuration items and assemble them into the interface parameters for calling the task scheduling platform, and create multiple scheduled tasks.

5. A dynamic data source switching system combined with a task scheduling platform, characterized in that Including a data source management module, a task management module, and a task execution management module, The data source management module manages data sources: Establish a data source information repository, and use the data source information repository to store the configuration information of the data sources, and initialize the data sources when the application service instance starts; The task management module performs task scheduling and management based on the task scheduling platform, stores task information in the database, and indicates the tenant code where the task is located in the task information. The tenant code is the same as the data source code. When the application service instance starts, register a scheduled task with the task scheduling platform. The task execution management module dynamically switches the data source configuration information according to the parameters of the scheduled task in the task information: Configure the interceptor of the task scheduling platform to intercept all scheduled task execution requests of the task scheduling platform through the interceptor. According to the scheduled task execution request, parse the task configuration information, obtain the data source code, and query the corresponding data source configuration information in the data source information library according to the data source code. Establish a connection with the target data source using the data source configuration information, and execute the corresponding scheduled task. After the scheduled task is completed, clear the data source information so that different data sources can be used for subsequent operations.

6. A dynamic data source switching system combined with a task scheduling platform according to claim 5, characterized in that the data source The management module also registers new data sources using the data source information library, modifies or deletes the configured data source information, and provides a data source query interface for querying the required data source configuration information before task execution. Introduce a template mechanism to create a data source configuration template, and define common connection parameters and advanced settings through the data source configuration template for scenarios where data sources need to be quickly deployed.

7. A dynamic data source switching system integrated with a task scheduling platform according to claim 5, wherein the data source The management module initializes the data source, including: Obtain all data configuration information. Traverse the data source configuration information. If it is empty, report an error. If it is not empty, generate a DataSource object according to the database connection string and connection pool parameter information in the data source configuration information. After initialization, store the DataSource objects of all data sources in the DynamicDataSourceContextHolder, a utility class for dynamically switching data sources.

8. A dynamic data source switching system integrated with a task scheduling platform according to claim 5, characterized in that The task management module registers a scheduled task with the task scheduling platform, including: Traverse all task lists to determine whether the tenant code where the task is located exists. If it exists, assemble the tenant code and the scheduling configuration items into the interface parameters for calling the task scheduling platform to create a scheduled task. The scheduling configuration items include scheduling type, executor id, routing policy, scheduling expiration policy, and blocking handling policy. If it does not exist, or the tenant code where it is located is "all", it means that the scheduled task needs to be executed by all tenants. Then obtain all data source codes, traverse the data source codes and the scheduling configuration items to assemble them into the interface parameters for calling the task scheduling platform to create multiple scheduled tasks.

Images