A lightweight multi-data source dynamic management method and system, a terminal device, and a computer readable storage medium

Abstract

The present application relates to a kind of lightweight multi-data source dynamic management method and system, terminal equipment, computer readable storage medium.The method includes the following steps: step S1: automatic configuration trigger and configuration loading;Step S2: data source initialization;Step S3: runtime dynamic configuration update;Step S4: data source switching and runtime management.The present application solves the problems of insufficient dynamicity, configuration update lag and operation and maintenance complexity in the prior art through automatic configuration trigger, dynamic data source instance generation, runtime real-time configuration update and multi-tenant connection isolation management, and has the advantages of improving data source dynamic management capability, supporting runtime real-time configuration update, simplifying multi-data source operation and maintenance complexity and enhancing system stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of databases, in particular to a lightweight multi-data source dynamic management method and system, a terminal device and a computer readable storage medium. BACKGROUND

[0002] In the development of enterprise-level applications based on Spring / SpringBoot, multi-data source dynamic integration is the core requirement for coping with scenarios such as database and table splitting, read-write separation, and multi-tenant architecture. The current mainstream technical solutions have significant limitations: static data source configuration solutions such as Spring AbstractRoutingDataSource have implemented basic routing functions, but the hard-coded data source mapping relationship makes it impossible to dynamically register or unregister data sources at runtime, especially in scenarios such as adding new tenant databases, which lack flexibility. More seriously, this solution cannot effectively integrate distributed transaction management when operating across data sources, forcing developers to manually implement compensation logic, significantly increasing system complexity and error risk. In addition, static solutions are difficult to adapt to the needs of database instance dynamic scaling in a cloud-native environment, and the root cause is the lack of a dynamic metadata management mechanism.

[0003] Middleware proxy solutions such as ShardingSphere-JDBC provide more comprehensive functions, but their architectural design has inherent flaws. The introduction of a proxy layer not only increases network latency and operational costs, but also causes data source change operations to be lagging, requiring a restart of the middleware or manual refresh for any configuration updates, which is contrary to the principles of modern DevOps agile development. More importantly, such solutions cannot directly reuse the transaction management and connection pool infrastructure in the Spring ecosystem, requiring additional development of an adaptation layer, resulting in a fragmented technology stack.

[0004] The common defects of existing technologies are concentrated in three aspects: first, the dynamicity is severely insufficient, unable to support real-time registration and destruction of data sources during application runtime; second, the transaction management capability is weak, making it difficult to achieve seamless integration with Spring declarative transactions; finally, the ecological compatibility is poor, unable to meet the standard data access needs of different persistence layer frameworks. These problems severely restrict the development efficiency and running stability of enterprise-level applications in complex data access scenarios.

[0005] In view of the above problems of the prior art, there is an urgent need to improve the prior art. SUMMARY

[0006] In view of the above defects of the prior art, the first aspect of the present application provides a lightweight multi-data source dynamic management method, which comprises the following steps:

[0007] Step S1: automatic configuration triggering and configuration loading; in the metadata storage layer, metadata information is stored and loaded through an SQLite embedded database, and the data source core parameters of the metadata information at least include one of the following: a data source identifier datasource_id, a database type db_type, a connection address endpoint, and authentication information credential_info; and a database driver is automatically derived and set according to the connection address;

[0008] Step S2: data source initialization; in the dynamic adaptation layer, a target data source instance is dynamically generated according to the data source core parameters, and the target data source instance is registered to an application running environment; wherein the target data source instance is a database connection pool object that is finally created and loaded into the application environment for business access;

[0009] Step S3: runtime dynamic configuration update; in the application running process, a standard SQL configuration statement input by a user is received, and data source configuration is added, deleted or modified in real time, and is synchronized to the data source instance in the application running environment;

[0010] Step S4: data source switching and runtime management; in the connection management layer, the target data source instance is executed for connection isolation, dynamic adjustment of the number of connections, heartbeat detection and failure transfer; at the same time, multi-tenant connection isolation is further supported to ensure that database access between different tenants is independent, and the maximum number of connections and the minimum number of idle connections of the connection pool are dynamically adjusted based on the runtime load.

[0011] In the lightweight multi-data source dynamic management method as described above, optionally, the step S1 includes the following sub-steps:

[0012] Step S1.1: starting the data source dynamic management module through a preset automatic configuration triggering mechanism;

[0013] Step S1.2: based on the automatic configuration assembly mechanism, a MonitorDbAutoConfiguration automatic database configuration component is taken as an assembly unit to realize automatic identification and loading of the database configuration class;

[0014] Step S1.3: the database management class under the predefined package scanning path com.cmonitor.api.db is identified through package scanning;

[0015] Step S1.4: in the configuration loading stage, the configuration parameters are uniformly loaded by using a DynDataSourceConfig dynamic data source configuration module;

[0016] Step S1.5: reading the data source configuration parameter with the cmonitor prefix from the external configuration environment;

[0017] Step S1.6: encapsulating the configuration parameter as a data source configuration object DataSourceVo for subsequent data source initialization.

[0018] In the lightweight multi-data source dynamic management method as described above, optionally, the step S2 includes the following sub-steps:

[0019] Step S2.1: loading the data source core parameter from the SQLite embedded database;

[0020] Step S2.2: creating a dynamic target data source instance based on the data source core parameter;

[0021] Step S2.3: selecting a default main data source from the plurality of candidate data sources and joining the mapping;

[0022] Step S2.4: selecting an initialization mode for configuration file initialization, the initialization mode at least including one of the following: database mode, configuration file mode.

[0023] In the lightweight multi-data source dynamic management method as described above, optionally, if the database mode is selected in the step S2.4 to initialize the configuration file, the following operations are performed: DDL table creation, query data source configuration, create DruidDataSource instance, verify and join the mapping.

[0024] In the lightweight multi-data source dynamic management method as described above, optionally, if the configuration file mode is selected in the step S2.4 to initialize the configuration file, the following operations are performed: traversing the configuration data source configuration object DataSourceVo, creating DruidDataSource instance, performing connection validity verification query validationQuery verification and joining the target data source instance targetDataSource mapping.

[0025] In the lightweight multi-data source dynamic management method as described above, optionally, the step S3 includes the following sub-steps:

[0026] Step S3.1: loading SqlSessionFactory and injecting the target data source instance;

[0027] Step S3.2: dynamically refreshing the mapping configuration of the mapping interface Mapper based on the target data source instance;

[0028] Step S3.3: When the user adds or modifies the data source configuration, the dynamic updating mechanism is used to refresh the database access path in real time.

[0029] In the lightweight multi-data source dynamic management method as described above, optionally, the step S4 comprises the following sub-steps:

[0030] Step S4.1: The data source context of the current thread is monitored by the dynamic data source context management tool DynDsUtil of the thread context monitoring mechanism, and at least one of the following management operations is performed: setDs, getDs, delDs; and the data source aspect DsAspect of the AOP aspect mechanism is used to intercept the data source annotation, wherein the priority of the annotation is method annotation, class annotation, and default annotation in sequence.

[0031] Step S4.2: When the routing key change is detected, the determineCurrentLookupKey method in DynDataSource is rewritten to determine the current target data source instance according to the context data source identifier.

[0032] Step S4.3: The target database availability is verified by the built-in heartbeat detection mechanism; when the connection failure is detected, the failover mechanism is automatically triggered to switch the data request to the standby candidate data source instance, and the current data source pointer is updated to maintain the continuity of the application access.

[0033] To achieve the above purpose, the second aspect of the present application provides a lightweight multi-data source dynamic management system, which uses the lightweight multi-data source dynamic management method as described in any one of the preceding first aspect embodiments, and is applied to a Spring application running environment, wherein it comprises:

[0034] A metadata storage module is realized by an SQLite embedded database and is configured to store and manage metadata information of the data source, wherein the metadata information at least includes data source identifier, database type, connection address, and authentication information.

[0035] A dynamic adaptation module is in communication connection with the metadata storage module and is configured to automatically trigger and load the configuration when the application is started; target data source instances are dynamically generated according to the metadata information obtained from the metadata storage module, and the target data source instances are registered to the Spring application context; wherein the dynamic adaptation module comprises a driver derivation unit for automatically deriving and setting the corresponding database driver according to the connection address.

[0036] A runtime management module, in communication connection with the metadata storage module and the dynamic adaptation module, is configured to receive a standard SQL configuration statement input by a user during application running, perform real-time adding, deleting and modifying operations on configuration data in the metadata storage module, and trigger the dynamic adaptation module to synchronously update data source instances in an application running environment;

[0037] A connection control module, in communication connection with the dynamic adaptation module, is configured to manage the target data source instances, and the management at least includes one of the following: multi-tenant based connection isolation, dynamic adjustment of connection number, heartbeat detection and failure switchover.

[0038] To achieve the above-mentioned purpose, the third aspect of the present application provides a terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the lightweight multi-data source dynamic management method according to any one of the preceding first aspect when running the program.

[0039] To achieve the above-mentioned purpose, the fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer executable instructions or a computer program, and the computer executable instructions or the computer program are executed by a processor to implement the lightweight multi-data source dynamic management method according to any one of the preceding first aspect.

[0040] The lightweight multi-data source dynamic management method and system, the terminal device and the computer readable storage medium provided by the present application solve the problems of insufficient dynamicity, lagging configuration update and complex operation and maintenance in the prior art through automatic configuration triggering, dynamic data source instance generation, real-time configuration update during runtime and multi-tenant connection isolation management, and have the advantages of improving data source dynamic management capability, supporting real-time configuration update during runtime, simplifying the complexity of multi-data source operation and maintenance and enhancing system stability.

[0041] The concept, specific structure and generated technical effects of the present application will be further described below with reference to the accompanying drawings, so as to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of an embodiment of the lightweight multi-data source dynamic management method provided by the present application;

[0043] Figure 2 is Figure 1 is a specific flowchart of an embodiment of step S1 in the method;

[0044] Figure 3 is Figure 1Specific flowchart of one embodiment in step S2;

[0045] Figure 4 is Figure 1 Specific flowchart of another embodiment in step S2;

[0046] Figure 5 is Figure 1 Specific flowchart of one embodiment in step S3;

[0047] Figure 6 is Figure 1 Flowchart of one embodiment in step S4. DETAILED DESCRIPTION

[0048] In order to make the technical means, creative features, purposes and effects of the application easy to understand, the application is further described below in combination with specific drawings. However, the application is not limited to the following embodiments.

[0049] It should be understood that the structures, proportions, sizes and the like shown in the drawings attached to the specification are only used to cooperate with the content disclosed in the specification for understanding and reading by those skilled in the art, and do not have technical substantive significance, and any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that can be achieved by the application, should still fall within the scope of the technical content disclosed by the application.

[0050] The terms such as "comprising" and "including" mean that the technical solution of the application does not exclude the presence of other components that are not directly or explicitly expressed in the specification and claims.

[0051] In addition, the term "and / or" in this paper only describes the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B, and the existence of B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the front and rear associated objects.

[0052] In the traditional existing multi-data source management scheme based on the Spring framework, the static data source configuration scheme depends on the pre-defined mapping relationship between the routing key and the data source, which causes the data source instance to be unable to be dynamically registered or destroyed at runtime. The middleware proxy scheme introduces an independent process layer to handle data source routing, which causes redundant architecture layering, causes the transaction management mechanism to be separated from the Spring native environment, lacks distributed transaction integration capability for cross-data source operations, and the data source change in the middleware layer needs to be manually refreshed, which complicates the operation and maintenance process.

[0053] For example, in an e-commerce platform using a microservice architecture, a separate database instance needs to be dynamically created for each new merchant. When using Spring AbstractRoutingDataSource, adding a new merchant data source requires modifying the routing mapping table in the code and redeploying the application, causing service interruptions. If ShardingSphere-JDBC middleware is used, adding a new data source requires configuring it through the proxy layer and restarting the middleware service. During this process, database requests need to wait for the configuration to take effect, increasing the average response latency to over 300 milliseconds. In multi-tenant order query scenarios, cross-merchant data source join queries lack deep integration with Spring declarative transactions, requiring manual writing of compensation logic to handle some failure operations, resulting in a transaction commit success rate dropping to 92.5%.

[0054] If the above issues are not addressed, the system will be unable to meet the dynamic scaling requirements of database instances in a cloud-native environment, hindering the elastic expansion of business scale. Network latency introduced by the middleware proxy layer directly impacts throughput metrics in high-concurrency scenarios, while fragmented transaction management mechanisms increase the risk of data inconsistency. Frequent service restarts caused by static configuration schemes reduce system availability, requiring the operations team to invest additional resources in maintaining data source configuration status, increasing manpower costs and the probability of operational errors.

[0055] like Figure 1 As shown, this application provides a lightweight multi-data source dynamic management method, which includes the following steps:

[0056] Step S1: Automatic configuration triggering and configuration loading.

[0057] In the metadata storage layer, metadata information is stored and loaded through an embedded SQLite database. The core parameters of the metadata data source include the data source identifier datasource_id, database type db_type, connection address endpoint, authentication information credential_info, etc.; and the database driver is automatically derived and set based on the connection address.

[0058] The metadata storage layer is a logical layer used to persistently store data source configuration parameters. It can be implemented using an embedded SQLite database and perform CRUD operations on metadata through a structured query language. This design allows changes to the data source configuration to take effect without restarting the application.

[0059] In optional embodiments, such as Figure 2 As shown, step S1 may include the following sub-steps:

[0060] Step S1.1: Start the data source dynamic management module through the preset automatic configuration trigger mechanism.

[0061] Step S1.2: Based on the automatic configuration assembly mechanism, the monitoring database automatic configuration component MonitorDbAutoConfiguration is assembled as an assembly unit to realize automatic identification and loading of the database configuration class.

[0062] Step S1.3: The database management class under the pre-defined package scanning path com.cmonitor.api.db is identified by scanning the package.

[0063] Step S1.4: In the configuration loading stage, the dynamic data source configuration module DynDataSourceConfig is used to uniformly load the configuration parameters.

[0064] Step S1.5: The data source configuration parameters with the cmonitor prefix are read from the external configuration environment.

[0065] Step S1.6: The configuration parameters are encapsulated as a data source configuration object DataSourceVo for subsequent data source initialization.

[0066] The automatic configuration trigger mechanism is configured to automatically activate the data source dynamic management function in response to the Spring Boot application startup event. For example, the automatic configuration trigger is based on the ApplicationPreparedEvent event of the Spring Boot framework, and the data source management module is initialized when the event is sensed. The monitoring database automatic configuration component MonitorDbAutoConfiguration is limited by the @ConditionalOnClass annotation to define the assembly condition, and the assembly process is activated when the DataSource class is detected to exist in the class path. The package scanning path is limited to com.cmonitor.api.db, and the scanning range is set by the basePackages attribute of the @ComponentScan annotation to avoid resource waste caused by scanning the full package path. The dynamic data source configuration module DynDataSourceConfig loads the configuration parameters in batches through the @ConfigurationProperties annotation, and the parameter prefix is set to cmonitor, such as cmonitor.datasource.url and cmonitor.datasource.username. The data source configuration object DataSourceVo is defined as a POJO class containing fields such as endpoint and credential_info, and is injected into the Spring context through the @Bean annotation.

[0067] Specifically, in the application startup phase, the automatic configuration trigger mechanism automatically activates the data source dynamic management module through event sensing, without the need for manual call of the initialization method. The MonitorDbAutoConfiguration database automatic configuration component automatically loads the predefined configuration class and registers it to the Spring container when detecting the existence of the database driver. The limitation of the package scanning path makes it necessary to only traverse the class files under the com.cmonitor.api.db path, reducing the file system scanning time by about 40%. The DynDataSourceConfig dynamic data source configuration module aggregates the scattered configuration parameters into a unified object in the loading phase, such as combining the 12 parameters with the cmonitor prefix into a structured configuration. The external parameters are read through the Environment interface and mapped to the fields of the DataSourceVo object, where the endpoint field of the connection address is automatically parsed by regular expression to determine the driver type, such as jdbc:mysql: / / being recognized as the MySQL driver, without the need for manual specification of the driver-class-name. The encapsulated DataSourceVo object exposes the parameters through the getter method for the data source instantiation phase, avoiding type conversion errors in the parameter transmission process.

[0068] Through the above technical solutions, the application realizes the automatic startup and configuration loading of the data source dynamic management module. The preset trigger mechanism eliminates the need for manual intervention, improving the reliability and efficiency of system initialization. The introduction of MonitorDbAutoConfiguration as an assembly unit realizes accurate identification and automatic loading of the database configuration class, reducing the possibility of configuration errors. The limited package scanning path improves the efficiency of configuration loading and avoids unnecessary resource consumption. The use of the DynDataSourceConfig module realizes unified management of configuration parameters, simplifying maintenance work. Reading parameters with a specific prefix from the external environment enhances the traceability and isolation of the configuration. Encapsulating the parameters into a structured DataSourceVo object provides a standardized data structure for subsequent data source initialization, improving the consistency and reliability of parameter transmission. These improvements collectively significantly improve the automation, accuracy, and efficiency of data source configuration management.

[0069] Step S2: Data source initialization.

[0070] In the dynamic adaptation layer, a target data source instance is dynamically generated according to the data source core parameters, and the target data source instance is registered to the application running environment. The target data source instance is the database connection pool object that is finally created and loaded into the application environment for business access.

[0071] The dynamic adaptation layer is an intermediate layer responsible for converting configuration parameters into available data source instances. Specifically, it can dynamically create a database connection pool object through a reflection mechanism and inject the instance into the application context using the Bean registration interface of the Spring framework to achieve runtime loading of the data source instance.

[0072] In an optional embodiment, step S2 can include the following sub-steps:

[0073] Step S2.1: Load the data source core parameters from the SQLite embedded database. The data source core parameters are dynamically loaded through the SQLite embedded database, and the parameter types can include connection addresses, authentication information, and other key metadata.

[0074] Step S2.2: Create a dynamic target data source instance based on the data source core parameters. During the dynamic instance creation process, the connection address can be parsed into a specific database driver type, for example, the driver class of MySQL can be automatically inferred based on the jdbc:mysql prefix.

[0075] Step S2.3: Select a default primary data source from multiple candidate data sources and add it to the mapping. The selection of the default primary data source is achieved through priority marking, and the candidate data source mapping table uses a hash table structure to store the primary and secondary relationships.

[0076] Step S2.4: Select an initialization method for configuration file initialization. The initialization method can include a database method or a configuration file method.

[0077] Optionally, the selection logic of the initialization method can be configured as an environment variable trigger, for example, the cloud environment prefers the database method, and the local deployment uses the configuration file method.

[0078] In this embodiment, if the database method is selected in step S2.4 to initialize the configuration file, as shown in Figure 3 the following operations are performed: DDL table creation, query data source configuration, create DruidDataSource instance, verify and add mapping.

[0079] In the DDL table creation operation, a configuration table can be generated by a pre-defined table structure template, which includes fields such as data source identifier, connection address, authentication information, etc. For example, the data source identifier can be stored in the form of VARCHAR(64), and the connection parameters in the form of JSON can be stored in the form of TEXT. When querying the data source configuration, a SELECT statement can be executed based on the data source identifier to obtain the configuration information, and a retry mechanism can be triggered when the query fails, for example, a query strategy with 3 retries and an interval of 500 milliseconds can be set. During the creation of DruidDataSource instance, connection pool parameters can be dynamically injected, for example, the initial connection number can be set to 5, the maximum connection number can be set to 20, and the connection leak detection function can be enabled. In the verification and mapping stage, the connection validity can be verified by executing the testOnBorrow configuration item, for example, SELECT 1 can be used as the verification query statement, and an alarm can be triggered and the initialization process can be terminated when the verification fails.

[0080] Specifically, in the database mode initialization process, first, a standardized DDL script is executed to create a configuration table, ensuring that the storage structure of the data source configuration information is consistent with the metadata definition, avoiding configuration loading exceptions caused by missing fields or type mismatches. Then, the configuration parameters are extracted from the database through a unified query interface, eliminating format errors that may be caused by manual input or file reading, for example, the JSON string returned by the database is deserialized into a DataSourceVo object. When creating DruidDataSource instance based on the configuration parameters, the initialization parameters of the connection pool are dynamically loaded from the database, for example, the validationQuery verification statement is automatically selected according to the database type, MySQL uses SELECT 1 and Oracle uses SELECT 1 FROM DUAL. Finally, before adding the data source instance to the target mapping table, the availability of the instance is confirmed by acquiring the connection and executing the verification query, for example, a 5-second timeout threshold is set, and if the timeout occurs, the instance is marked as unavailable and an error log is recorded. Through the standardized design of the above process and the effectiveness verification mechanism, it is ensured that the data source instance has complete structure constraints and running reliability during dynamic generation, effectively avoiding system abnormalities caused by incomplete configuration information or connection parameter errors.

[0081] In this embodiment, if the configuration file mode is selected in step S2.4 to perform configuration file initialization, as shown in Figure 4 the following operations are performed: traversing the configuration data source configuration object DataSourceVo, creating DruidDataSource instance, executing connection validity verification query validationQuery verification, and adding target data source instance targetDataSource mapping.

[0082] The implementation of traversing the configuration data source configuration object DataSourceVo includes accessing the configuration items one by one through an iterator or a loop structure, and ensuring that all parameters with the cmonitor prefix are processed. For example, the configuration items can include a connection address, authentication information, and a timeout parameter, and the corresponding parameters are extracted through key-value matching during the traversal process. When creating a DruidDataSource instance, the createDataSource method of DruidDataSourceFactory is called, and the configuration parameters are passed in to build a standardized connection pool object. The connection pool initialization parameters can include initialSize=5 and maxActive=20. When performing a connection validity verification query, the validationQuery is set to a test statement related to the database type, for example, SELECT 1 is used for MySQL, and the validation timeout time can be set to 5 seconds. If the verification fails, an exception is thrown and the instance registration is terminated.

[0083] Specifically, during the configuration file initialization process, all configuration items of the DataSourceVo object are first traversed, and the configuration parameters are injected into the DruidDataSource instance through a reflection mechanism or attribute mapping. For example, when the configuration item includes maxWait=6000, the parameter is automatically set to the maximum wait time attribute of the connection pool. Subsequently, a test request is sent to the database by executing the validationQuery, and if no response is received within the specified time, the connection is determined to be invalid and the instance is discarded. The verified instance is added to the targetDataSource mapping table, and the mapping key is the data source identifier, for example, the tenant ID or the business module name is used as the key value. In this way, the configuration loading integrity is guaranteed through the traversal mechanism, and the connection validity is controlled through the active verification mechanism, and the two mechanisms work together to avoid the problem of invalid connection registration caused by configuration omission or network fluctuations, thereby improving the availability of the data source instance.

[0084] In summary, in the data source initialization phase, first, the core parameter set is extracted from the SQLite database, and the connection address is converted into the corresponding driver class instance through the parameter parsing engine. Then, the data source connection pool object is constructed according to the parameter set, which automatically performs connection validity verification when created. In the candidate data source registration process, the system selects the first data source with the primary identification bit as the default primary data source by traversing the loaded data source instances, and writes its mapping relationship into the global routing table. The selection of the initialization method is realized through the environment detection module, and when the external configuration center service is detected to be available, the database method is automatically enabled to perform the DDL table creation operation; when the local configuration file is detected to exist, the configuration object is loaded through the deserialization mechanism. This process realizes multi-source adaptation through the dynamic parameter loading mechanism, eliminates routing conflicts through the primary-secondary mapping relationship, and is compatible with different deployment environments through multi-mode initialization.

[0085] Through the above technical solutions, the application realizes the dynamics and flexibility of data source initialization. By dynamically loading core parameters from the SQLite database, hard-coded configuration is avoided, and the maintainability of the system is improved. Based on dynamic parameters, data source instances are created, so that the system can adapt to different types of databases, enhancing compatibility. The default primary data source is specified and the mapping relationship is established, solving the routing conflict problem in the multi-data source environment and ensuring the stability of data access. Support for multiple initialization methods meets the standardization needs and retains the possibility of flexible configuration, adapting to different deployment scenarios. These improvements collectively improve the system's adaptability and running efficiency in complex multi-source environments.

[0086] Step S3: runtime dynamic configuration update.

[0087] During application running, standard SQL configuration statements input by the user are received, and data source configuration is added, deleted or modified in real time, and is synchronized to the data source instance in the application running environment.

[0088] Among them, the standard SQL configuration statement refers to a database operation instruction conforming to the ANSI SQL specification, which can specifically include data manipulation statements such as INSERT, UPDATE, DELETE, etc. By parsing and executing these statements, the configuration records in the metadata storage layer are directly modified, realizing real-time effectiveness of configuration changes.

[0089] In an optional embodiment, as shown in Figure 5 the step S3 can include the following sub-steps:

[0090] Step S3.1: Load SqlSessionFactory and inject target datasource instance. The loading of SqlSessionFactory can be achieved by the initialized Bean instance in Spring container, and the injection of target datasource instance can be achieved by replacing the original datasource reference, for example, using dynamic proxy mechanism to bind the newly generated datasource object to the configuration property of SqlSessionFactory.

[0091] Step S3.2: Dynamically refresh the mapping configuration of the mapping interface Mapper based on the target datasource instance. The dynamic refresh of mapping configuration can be achieved by re-parsing the Mapper XML file or updating the interface proxy object, for example, calling the addMapper method of configuration to re-register the Mapper interface, and clearing the original mapping cache.

[0092] Step S3.3: When the user adds or modifies the datasource configuration, use the dynamic update mechanism to refresh the database access path in real time. The trigger of dynamic update mechanism can be achieved by sensing the configuration change event, for example, publishing DataChangeEvent to event bus after updating datasource parameters, automatically capturing events, responding and triggering refresh process. The binding relationship between datasource instance and Mapper interface is maintained through thread local variable, for example, associating datasource_id in current thread context with the namespace of Mapper interface, ensuring that SQL execution is automatically routed to the correct datasource.

[0093] Specifically, during the loading process of SqlSessionFactory, the dynamically generated target data source instance is injected into the dataSource field in the environment attribute by obtaining the internal configuration object, so that the MyBatis execution engine can directly access the updated connection pool. When the Mapper mapping configuration needs to be refreshed, the addMapper method of the configuration is called one by one to reload the interface definition by traversing the registered Mapper interface list, and the MappedStatement object in the internal cache is cleared to ensure that the SQL statement is associated with the new data source instance. After the user modifies the data source configuration, the dynamic update mechanism triggers the recreation of the data source instance and the batch refresh of the Mapper interface by intercepting the configuration change operation, for example, when the endpoint parameter is changed, a new DruidDataSource instance is automatically generated and replaces the original reference, and then the addMappers method of MapperRegistry is called to update the interface binding relationship. Through the linkage execution of the above steps, the real-time synchronization of the data source instance and the persistence layer configuration is realized, and the service interruption or manual restart operation caused by configuration update is avoided.

[0094] Through the above technical solutions, the application realizes the deep integration and real-time synchronization of the data source configuration and the persistence layer framework. Therefore, the service interruption or manual restart problem caused by data source change in the traditional scheme is avoided. Specifically, through the dynamic injection and refresh mechanism, the application can seamlessly switch the data source at runtime while maintaining compatibility with the ORM framework. This method not only improves the usability and flexibility of the system, but also greatly reduces the operation and maintenance complexity. Further, the real-time refresh mechanism ensures the consistency and accuracy of data access, effectively preventing data errors or exceptions that may be caused by asynchronous configuration.

[0095] Step S4: data source switching and runtime management.

[0096] In the connection management layer, connection isolation, dynamic adjustment of connection number, heartbeat detection and failure transfer are performed on the target data source instance. At the same time, multi-tenant connection isolation is further supported to ensure that database access between different tenants is independent, and the maximum number of connections and the minimum number of idle connections of the connection pool are dynamically adjusted based on the runtime load.

[0097] Among them, the connection management layer refers to a functional module for runtime monitoring and scheduling of data source connections. Specifically, multi-tenant connection isolation can be achieved through thread isolation technology, and the maximum connection number configuration is automatically optimized based on the load situation combined with the connection pool parameter dynamic adjustment algorithm to ensure resource utilization in high concurrency scenarios.

[0098] In optional embodiments, as shown in Figure 6 Step S4 can include the following sub-steps:

[0099] Step S4.1: Monitor the data source context of the current thread by the dynamic data source context management tool DynDsUtil of the thread context monitoring mechanism, and perform at least one of the following management operations: setDs, getDs, delDs; and intercept the data source annotation by the data source aspect DsAspect of the AOP aspect mechanism, wherein the priority of the annotation is in the order of method annotation, class annotation, and default annotation.

[0100] Step S4.2: When the routing key change is detected, rewrite the determineCurrentLookupKey method in DynDataSource to determine the current target data source instance according to the context data source identifier.

[0101] Step S4.3: Verify the availability of the target database by the built-in heartbeat detection mechanism; when the connection failure is detected, automatically trigger the failover mechanism to switch the data request to the standby candidate data source instance, and update the current data source pointer to maintain the continuity of application access.

[0102] The dynamic data source context management tool can maintain the data source identifier by thread local storage variables, for example, store the key-value mapping relationship by using the ThreadLocal data structure, allow the datasource_id of the current thread to be set by the setDs operation, and the data source identifier bound to the current thread to be obtained by the getDs operation. The AOP aspect mechanism scans the annotation hierarchy before executing the target method, preferentially parses the @DataSource annotation at the method level, if it does not exist, then looks up the class-level annotation, and finally uses the default data source identifier, thereby eliminating the routing ambiguity caused by multiple layers of annotation coverage. The dynamic lookup key method actively extracts the latest data source identifier from the thread context when detecting the routing key change event, and quickly locates the corresponding connection pool instance based on the hash table, for example, uses ConcurrentHashMap to store the mapping relationship between the data source identifier and the instance, and the query time complexity is O(1). The heartbeat detection mechanism sends a probe request to the target database at a fixed frequency, for example, performs a SELECT 1 query every 30 seconds, if the probe is timed out for 3 times in succession, it is determined that the connection is failed, the redirection process is triggered to redirect the request to the predefined standby instance, and the global data source pointer is updated to point to the new instance address.

[0103] Specifically, when the business method is called, the data source aspect first intercepts the method execution, parses the target data source identifier according to the annotation priority, and binds the identifier to the current thread through a dynamic data source context management tool. During the execution of the data access operation, the dynamic data source obtains the thread-bound identifier in real time through the rewritten lookup key method, so as to accurately route to the corresponding database instance. If the primary data source is unavailable due to network failure at this time, the heartbeat detection module identifies the abnormal state, and automatically switches the subsequent request to the standby instance, for example, switches the original request pointing to db_primary to db_secondary, and records the transfer event through the log after the switching is completed. The whole process ensures that the data source identifiers do not interfere with each other in the multi-tenant scenario through thread-isolated context management, avoids routing conflicts based on the clear annotation interception mechanism with priority, and guarantees the high availability of the system in combination with the active heartbeat detection and automatic failure transfer.

[0104] Through the above technical solution, the application realizes fine data source context management, and solves the problem of data source switching chaos in the multi-tenant scenario. By defining clear annotation priority, the data source routing decision conflict is eliminated. At the same time, the heartbeat detection and failure transfer mechanism improves the availability of the system, and reduces the risk of application access interruption caused by database connection failure. This dynamic management mode enhances the flexibility and reliability of the system, making the data source switching more efficient and stable.

[0105] Overall, the application realizes dynamic registration and destruction of data sources, and can complete the update of data source configuration without restarting the application. The system supports adding new data source instances at runtime, adapting to the needs of database dynamic scaling in a cloud environment. By integrating data source management logic into the application runtime environment, additional network delay is avoided, and the system response speed is improved. The multi-tenant connection isolation mechanism ensures the security of different tenant data, and the dynamic adjustment of connection pool parameters optimizes the resource utilization efficiency. The heartbeat detection and failure transfer function enhances the availability of the system, and reduces the service interruption caused by database connection problems. The overall scheme reduces the operation and maintenance complexity, improves the scalability and stability of the system.

[0106] In order to achieve the above purpose, the application also provides a lightweight multi-data source dynamic management system, wherein the lightweight multi-data source dynamic management method described in any one of the preceding embodiments is applied in a Spring application runtime environment, and specifically can include a metadata storage module, a dynamic adaptation module, a runtime management module, and a connection control module.

[0107] Specifically, the metadata storage module can be implemented by an SQLite embedded database, and is configured to store and manage metadata information of the data source, which can include data source identification, database type, connection address, authentication information, and the like.

[0108] The dynamic adaptation module is in communication connection with the metadata storage module, and is configured to automatically trigger and load the configuration when the application is started. The target data source instance is dynamically generated according to the metadata information obtained from the metadata storage module, and the target data source instance is registered to the Spring application context. The dynamic adaptation module includes a driver derivation unit, which is used to automatically derive and set the corresponding database driver according to the connection address.

[0109] The runtime management module is in communication connection with the metadata storage module and the dynamic adaptation module, and is configured to receive the standard SQL configuration statement input by the user during the application running process, perform real-time adding, deleting, and modifying operations on the configuration data in the metadata storage module, and trigger the dynamic adaptation module to synchronously update the data source instance in the application running environment.

[0110] The connection control module is in communication connection with the dynamic adaptation module, and is configured to manage the target data source instance, which can include connection isolation based on multi-tenancy, dynamic adjustment of the number of connections, heartbeat detection, and failure transfer, and the like. The specific embodiments have been described in detail above, and will not be described again.

[0111] To achieve the above purpose, the application further provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the lightweight multi-data source dynamic management method as described in any one of the preceding embodiments when running the program. The processor and the memory can be separately arranged or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0112] To achieve the above purpose, the application further provides a computer readable storage medium, which stores computer executable instructions or a computer program, and the computer executable instructions or the computer program are executed by a processor to implement the lightweight multi-data source dynamic management method as described above.

[0113] The computer readable storage medium is, for example, a memory. The memory can be a volatile memory or a non-volatile memory, or the memory can include both volatile and non-volatile memory. The non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory, for example. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available, for example, a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), and a direct Rambus RAM (DRRAM).

[0114] The integrated units in the above embodiments, if implemented in the form of software functional units and sold or used as independent products, can be stored in the above computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0115] The preferred embodiments of the present application are described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations to the present application without requiring inventive effort. Therefore, the scope of the present application should be determined not by the preferred embodiments, but by the appended claims and their equivalents.

Claims

1. A lightweight multi-data source dynamic management method, characterized in that, Comprise the following steps: Step S1: automatic configuration trigger and configuration loading; in the metadata storage layer, the metadata information is stored and loaded by SQLite embedded database, and the data source core parameters of the metadata information at least include one of the following: data source identification datasource_id, database type db_type, connection address endpoint, authentication information credential_info; and automatically derive and set the database driver according to the connection address; Step S2: data source initialization; in the dynamic adaptation layer, the target data source instance is dynamically generated according to the data source core parameters, and the target data source instance is registered to the application running environment; wherein the target data source instance is the database connection pool object finally created and loaded into the application environment for business access; Step S3: runtime dynamic configuration update; in the application running process, the standard SQL configuration statement input by the user is received, the data source configuration is increased, deleted or modified in real time, and is synchronized to the data source instance in the application running environment; Step S4: data source switching and runtime management; in the connection management layer, the target data source instance is executed to perform connection isolation, dynamic adjustment of connection number, heartbeat detection and failure transfer; at the same time, further support is provided for multi-tenant connection isolation to ensure that the database access between different tenants is independent of each other, and the maximum connection number and the minimum idle connection number of the connection pool are dynamically adjusted based on the runtime load.

2. The lightweight multi-data source dynamic management method of claim 1, wherein, The step S1 comprises the following substeps: Step S1.1: starting the data source dynamic management module through the preset automatic configuration trigger mechanism; Step S1.2: based on the automatic configuration assembly mechanism, the monitoring database automatic configuration component MonitorDbAutoConfiguration is taken as an assembly unit to realize automatic identification and loading of the database configuration class; Step S1.3: the database management class under the pre-defined package scanning path com.cmonitor.api.db is identified by scanning the package; Step S1.4: in the configuration loading stage, the dynamic data source configuration module DynDataSourceConfig is used to uniformly load the configuration parameters; Step S1.5: reading the data source configuration parameters with cmonitor prefix from the external configuration environment; Step S1.6: the configuration parameters are packaged into the data source configuration object DataSourceVo for subsequent data source initialization.

3. The lightweight multi-data source dynamic management method of claim 1, wherein, The step S2 comprises the following substeps: Step S2.1: loading the data source core parameters from the SQLite embedded database; Step S2.2: creating a dynamic target data source instance based on the data source core parameters; Step S2.3: selecting a default primary data source from a plurality of candidate data sources and adding mapping; Step S2.4: selecting an initialization mode for configuration file initialization, and the initialization mode at least includes one of the following: database mode, configuration file mode.

4. The lightweight multi-data source dynamic management method of claim 3, wherein, If the database mode is selected in the step S2.4 to initialize the configuration file, the following operations are performed: DDL table creation, query data source configuration, creation of DruidDataSource instance, verification and addition of mapping.

5. The lightweight multi-data source dynamic management method of claim 3, wherein, If the configuration file mode is selected in the step S2.4 to initialize the configuration file, the following operations are performed: traversal of configuration data source configuration object DataSourceVo, creation of DruidDataSource instance, execution of connection validity verification query validationQuery verification and addition of target data source instance targetDataSource mapping.

6. The lightweight multi-data source dynamic management method of claim 1, wherein, The step S3 comprises the following sub-steps: Step S3.1: loading SqlSessionFactory and injecting the target data source instance; Step S3.2: dynamically refreshing the mapping configuration of the mapping interface Mapper based on the target data source instance; Step S3.3: when the user adds or modifies the data source configuration, the dynamic updating mechanism is used to refresh the database access path in real time.

7. The lightweight multi-data source dynamic management method of claim 1, wherein, The step S4 comprises the following sub-steps: Step S4.1: monitoring the data source context of the current thread by the dynamic data source context management tool DynDsUtil of the thread context monitoring mechanism, and performing at least one of the following management operations: setDs, getDs, delDs; And intercepting the data source annotation by the data source aspect DsAspect of the AOP aspect mechanism, wherein the priority of the annotation is method annotation, class annotation, and default annotation in sequence; Step S4.2: when the routing key is detected to change, the determineCurrentLookupKey method in DynDataSource is rewritten to determine the current target data source instance according to the context data source identifier; Step S4.3: verifying the availability of the target database by the built-in heartbeat detection mechanism; when the connection is detected to be invalid, the invalid transfer mechanism is automatically triggered to switch the data request to the backup candidate data source instance, and the current data source pointer is updated to maintain the continuity of application access.

8. A lightweight multi-data source dynamic management system using the lightweight multi-data source dynamic management method according to any one of claims 1 to 7, applied to a Spring application running environment. It comprises: A metadata storage module, which is realized by an SQLite embedded database and is configured to store and manage metadata information of the data source, the metadata information at least including data source identifier, database type, connection address and authentication information; A dynamic adaptation module, which is in communication connection with the metadata storage module and is configured to automatically trigger and load the configuration when the application starts; dynamically generating a target data source instance according to the metadata information obtained from the metadata storage module, and registering the target data source instance to the Spring application context; wherein the dynamic adaptation module comprises a driver derivation unit for automatically deriving and setting the corresponding database driver according to the connection address. A runtime management module, in communication connection with the metadata storage module and the dynamic adaptation module, is configured to receive a standard SQL configuration statement input by a user during application running, perform real-time adding, deleting, and modifying operations on configuration data in the metadata storage module, and trigger the dynamic adaptation module to synchronously update data source instances in an application running environment; A connection control module, in communication connection with the dynamic adaptation module, is configured to manage the target data source instances, and the management at least includes one of the following: multi-tenant-based connection isolation, dynamic adjustment of the number of connections, heartbeat detection, and failover.

9. A terminal device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the lightweight multi-data-source dynamic management method in any one of claims 1 to 7 when running the program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions or a computer program, and the computer executable instructions or the computer program are executed by a processor to implement the lightweight multi-data-source dynamic management method in any one of claims 1 to 7.

Images