Cache-based multi-source data management method and system

Abstract

The application discloses a multi-source data management method and system based on a cache, belongs to the technical field of data management, and aims to solve the technical problem of how to ensure that an application system is quickly connected with a data source under the premise of reducing system energy consumption.The method comprises the following steps: cache initialization: configuring basic parameters of the cache to form a configuration file, initializing the local cache through the configuration file, storing a data source connection pool in the cache, configuring a listening service for the cache, and listening to the connection and use state of the data source connection pool through the cache listening service; cache addition: during the process of processing data by a business system, a newly-built data source is added into the cache based on data source information; and cache removal: removing the data source connection pool in the cache.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data governance, in particular to a multi-source data management method and system based on cache. BACKGROUND

[0002] How to access various types of data sources quickly becomes the key to data governance. In the process of data governance, multiple types of data sources are usually involved, and there is more than one data source for each type. In the traditional database connection mode, each data access must go through the steps of establishing a database connection, opening a database, accessing data, and closing a database connection. Database connection is a critical, limited and expensive resource. Frequent connection and disconnection increase the overhead of repeated resource consumption, which greatly reduces the performance of the system. Therefore, the use of connection has become a bottleneck of system performance, especially in today's era of application concurrency and massive data, this performance bottleneck is particularly prominent.

[0003] In the face of more and more data lakes or data governance platforms connected to external data sources, how to ensure that other applications can quickly interface with data sources, reduce personnel development costs, reduce system resource consumption, and monitor the connection and use status of data sources in real time is a technical problem that needs to be solved. SUMMARY

[0004] The technical task of the present application is to solve the technical problem of how to ensure that the application system quickly interfaces with the data source under the premise of reducing system energy consumption in view of the above shortcomings, and provide a multi-source data management method and system based on cache.

[0005] In a first aspect, the present application provides a multi-source data management method based on cache, which stores a data source connection pool in cache. The method comprises the following steps:

[0006] Cache initialization: configure the basic parameters of the cache to form a configuration file, read the configuration file when the business system starts, and initialize the local cache through the configuration file. The cache is used to store the data source connection pool, and a listening service is configured for the cache to listen to the connection and use status of the data source connection pool;

[0007] Cache addition: during the data processing of the business system, for a newly created data source, based on the data source information, the data source connection pool corresponding to the data source is put into the cache;

[0008] Cache removal: remove the data source connection pool in the cache.

[0009] As a preferred, the listening service comprises:

[0010] Cache monitoring, which is used to monitor the number of data source connection pools in the cache, the time when the data source connection pool is added to the cache, the invalidation time, and the state of the data source connection pool;

[0011] Cache invalidation monitoring, which is triggered when the data source connection pool is removed, and is used to close the data source connection pool;

[0012] Data source invalidation monitoring, which is used to determine whether there is a data source connection pool corresponding to the data source in the cache when the data source itself is deleted by the business system, and if so, the data source connection pool is removed from the cache.

[0013] Preferably, the basic parameters of the cache include cache initialization capacity, cache maximum capacity, eviction policy, and expiration time;

[0014] The eviction policy is that when a new data source connection pool is created but the current cache capacity reaches the upper limit of the capacity, the eviction policy is triggered, and for the data source connection pool with the least usage in the current cache, the data source connection pool is removed from the cache, and the newly created data source connection pool is stored in the cache;

[0015] The expiration time is a predetermined period of time during which the data source connection pool in the cache is not used, and when the data source connection pool in the cache is not used for more than the predetermined period of time, the data source connection pool is considered as an expired data source connection pool;

[0016] The data source connection pool in the cache is removed, including the following three cases:

[0017] Cache expiration removal: when the unused time of the data source connection pool in the cache exceeds the expiration time, the data source connection pool is determined to be expired, and the data source connection pool is removed from the cache;

[0018] Eviction policy removal: when a new data source connection pool is created, if the current cache capacity reaches the maximum cache capacity, the eviction policy is triggered, and for the data source connection pool in the cache that exceeds the least usage, the data source connection pool is removed from the cache;

[0019] Removal of removal messages: after receiving a cache removal message, for the data source connection pool specified in the cache removal message, the data source connection pool is removed from the cache.

[0020] Preferably, for a newly created data source, based on the data source information, the data source connection pool corresponding to the data source is put into the cache, including the following steps:

[0021] a. The user creates or maintains new data source connection information;

[0022] b. After the business system acquires the new data source connection information, the data source connection information is tested, if the test is successful, step c is executed, otherwise, step a is returned to modify the data source connection information again;

[0023] c. A unique identifier is generated based on the data source connection information through the MD5 algorithm, the unique identifier judges whether the cache exists the data source connection pool, if not, step d is executed, if exists, the cache is added;

[0024] d. According to the data source information, the corresponding data source connection pool is created, and it is judged whether the current cache capacity reaches the maximum cache capacity, if yes, the elimination strategy is triggered, based on the elimination strategy, the data source connection pool with the least use frequency in the cache is removed from the cache, and the newly created data source connection pool is stored in the cache.

[0025] Among them, the data source connection information includes data source connection address, data source account, database name and database version number.

[0026] As preferred, the data source connection pool in the cache is removed, including the following steps:

[0027] a. It is judged whether there is still an active connection being used in the data source connection pool to be removed, if yes, step b is executed, otherwise, step c is executed;

[0028] b. The data source connection pool is removed from the cache, after the active connection completes the work, all connections in the data source connection pool are in idle state, all connections in the data source connection pool are closed, and the data source connection pool is destroyed;

[0029] c. All connections in the data source connection pool are closed and the data source connection pool is destroyed, and the data source connection pool is removed from the cache.

[0030] In the second aspect, the application is a cache-based multi-source data management system for executing the cache-based multi-source data management method as any one of the first aspect, the system comprises:

[0031] The cache initialization module is used to execute: configuring the basic parameters of the cache to form a configuration file, reading the configuration file when the business system starts and initializing the local cache through the configuration file, the cache is used to store the data source connection pool, and the cache is configured to listen to the service, the connection and use state of the data source connection pool are listened to through the cache listening service;

[0032] The cache adding module is used to execute: during the data processing of the business system, for the newly created data source, based on the data source information, the data source connection pool corresponding to the data source is put into the cache;

[0033] a cache removal module, configured to perform removal of the data source connection pool in the cache.

[0034] Preferably, the monitoring service comprises:

[0035] cache monitoring, configured to monitor the number of the data source connection pool in the cache, the time when the data source connection pool is added to the cache, the invalidation time, and the state of the data source connection pool;

[0036] cache invalidation monitoring, triggered when the data source connection pool is removed, configured to close the data source connection pool;

[0037] data source invalidation monitoring, configured to determine whether there is a data source connection pool corresponding to the data source in the cache when the data source itself is deleted by the business system, and remove the data source connection pool from the cache if there is.

[0038] Preferably, the basic parameters of the cache comprise cache initialization capacity, cache maximum capacity, eviction policy, and expiration time.

[0039] The eviction policy is that when a new data source connection pool is created but the capacity of the current cache reaches the upper limit of the capacity, the eviction policy is triggered, and for the data source connection pool with the least usage frequency in the current cache, the data source connection pool is removed from the cache, and the newly created data source connection pool is stored in the cache.

[0040] The expiration time is a predetermined period of time during which the data source connection pool in the cache is not used, and when the data source connection pool in the cache is not used for more than the predetermined period of time, the data source connection pool is considered as an expired data source connection pool.

[0041] The removal of the data source connection pool in the cache comprises the following three cases:

[0042] cache expiration removal: when the non-use time of the data source connection pool in the cache exceeds the expiration time, it is determined that the data source connection pool is expired, and for the expired data source connection pool, the data source connection pool is removed from the cache;

[0043] eviction policy removal: when a new data source connection pool is created, if the capacity of the current cache reaches the maximum value of the cache capacity, the eviction policy is triggered, and for the data source connection pool in the cache that exceeds the least usage frequency, the data source connection pool is removed from the cache;

[0044] removal message removal: after receiving a cache removal message, for the data source connection pool specified in the cache removal message, the data source connection pool is removed from the cache.

[0045] As preferred, the cache adding module is used to implement putting the data source connection pool corresponding to the data source into the cache based on the data source information as follows:

[0046] a. A user creates or maintains new data source connection information;

[0047] b. After a business system acquires the new data source connection information, the data source connection information is tested. If the test is successful, step c is executed, otherwise, step a is returned to modify the data source connection information again;

[0048] c. A unique identifier is generated based on the data source connection information through MD5 algorithm. The unique identifier judges whether the data source connection pool exists in the cache. If not, step d is executed, if yes, the cache adding is ended;

[0049] d. According to the data source information, the corresponding data source connection pool is created, and it is judged whether the capacity of the current cache reaches the maximum capacity of the cache. If yes, the eviction policy is triggered. Based on the eviction policy, the data source connection pool with the least usage frequency in the cache is removed from the cache, and the newly created data source connection pool is stored in the cache;

[0050] Wherein, the data source connection information includes data source connection address, data source account, database name, and database version number.

[0051] As preferred, the cache removing module is used to implement removing the data source connection pool in the cache as follows:

[0052] a. It is judged whether there is still an active connection being used in the data source connection pool to be removed. If yes, step b is executed, otherwise, step c is executed;

[0053] b. The data source connection pool is removed from the cache. After the active connection completes the work, when all the connections in the data source connection pool are in idle state, all the connections in the data source connection pool are closed, and the data source connection pool is destroyed;

[0054] c. All the connections in the data source connection pool are closed, the data source connection pool is destroyed, and the data source connection pool is removed from the cache.

[0055] Wherein, removing the data source connection pool in the cache includes the following three cases:

[0056] Cache expiration removal: the data source connection pool in the cache which is not used for more than a predetermined period of time is regarded as an expired data source connection pool. The data source connection pool is removed from the cache;

[0057] Eviction policy triggers removal: when a new data source connection pool is created, but the current cache capacity reaches the upper limit of the capacity, the eviction policy is triggered, and the data source connection pool with the least usage frequency in the current cache is removed from the cache based on the eviction policy, and the newly created data source connection pool is stored in the cache.

[0058] Message-based removal: based on the received cache removal message, the data source connection pool specified in the cache removal message is removed from the cache.

[0059] The cache-based multi-source data management method and system of the present application has the following advantages:

[0060] 1. Compared with the traditional real-time connection method, the resource reuse degree is improved, and the resource consumption is reduced.

[0061] 2. By using cache to manage data source connection pool, the difficulty of multi-data source connection pool management is reduced by using the characteristics of cache itself.

[0062] 3. Platform independence: not dependent on any specific database, so it can be completely heterogeneous. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] The present application will be further described below in conjunction with the drawings.

[0065] Figure 1 The flow chart of the cache-based multi-source data management method of embodiment 1. DETAILED DESCRIPTION

[0066] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it. However, the embodiments are not limiting to the present application, and the technical features in the embodiments and the embodiments can be combined with each other without conflict.

[0067] The present application provides a cache-based multi-source data management method and system to solve the technical problem of how to ensure that the application system quickly interfaces with the data source under the premise of reducing system energy consumption.

[0068] Embodiment 1:

[0069] The application discloses a multi-source data management method based on a cache.

[0070] Step S100 of cache initialization: basic parameters of the cache are configured to form a configuration file, the configuration file is read when a business system is started, and the local cache is initialized through the configuration file, the cache is used for storing a data source connection pool, and a listening service is configured for the cache, and the connection and use state of the data source connection pool are listened to through the cache listening service.

[0071] In this step, the basic parameters of the cache include a cache initialization capacity, a cache maximum capacity, an elimination strategy and an expiration time, the basic parameters are maintained in the configuration file, and the local cache is initialized by reading the configuration file when the application system is started, and the cache is used for storing data source connection pool information.

[0072] The elimination strategy is that when a new data source connection pool is created and the capacity of the current cache reaches the capacity upper limit, the elimination strategy is triggered, the data source connection pool with the least use frequency in the current cache is removed from the cache, and the new data source connection pool is stored in the cache.

[0073] The expiration time is a predetermined period of time during which the data source connection pool in the cache is not used, and when the data source connection pool in the cache is not used for more than the predetermined period of time, the data source connection pool is regarded as an expired data source connection pool.

[0074] Meanwhile, the cache needs to be configured with a listening service, and the listening service includes three services of cache listening, cache invalidation listening and data source invalidation listening.

[0075] The cache listening is used for listening to the number of data source connection pools in the cache, the time when the data source connection pool is added to the cache, the invalidation time of the data source connection pool and the state of the data source connection pool.

[0076] The cache invalidation listening is triggered when the data source connection pool is removed, and is used for closing the data source connection pool to avoid resource consumption caused by memory leakage.

[0077] When the data source itself is deleted by the business system, the data source invalidation listening is used for judging whether the data source connection pool corresponding to the data source exists in the cache, and if the data source connection pool exists, the data source connection pool is removed from the cache.

[0078] Step S200 of cache addition: for a newly created data source, the data source connection pool corresponding to the data source is put into the cache based on the data source information during the data processing of the business system.

[0079] In this step, for the newly created data source, based on the data source information, the data source connection pool corresponding to the data source is put into the cache, including the following steps:

[0080] a. A user creates or maintains new data source connection information;

[0081] b. After the business system obtains the new data source connection information, the data source connection information is tested. If the test is successful, step c is executed, otherwise, step a is returned to modify the data source connection information again;

[0082] c. Based on the data source connection information, a unique identifier is generated through the MD5 algorithm. The unique identifier judges whether the data source connection pool exists in the cache. If it does not exist, step d is executed, and if it exists, the cache addition is ended;

[0083] d. According to the data source information, the corresponding data source connection pool is created, and it is judged whether the capacity of the current cache reaches the maximum capacity of the cache. If it does, the eviction policy is triggered, and based on the eviction policy, the data source connection pool with the least usage frequency in the cache is removed from the cache, and the newly created data source connection pool is stored in the cache.

[0084] Among them, the data source connection information includes data source connection address, data source account, database name, and database version number.

[0085] Step S300 cache removal: removing the data source connection pool in the cache.

[0086] In this embodiment, the data source connection pool in the cache is removed, including cache expiration removal, eviction policy removal, and removal message removal.

[0087] Cache expiration removal: When the unused time of the data source connection pool in the cache exceeds the expiration time, it is determined that the data source connection pool is expired, and the data source connection pool is removed from the cache.

[0088] Eviction policy removal: When a new data source connection pool is created, if the capacity of the current cache reaches the maximum capacity of the cache, the eviction policy is triggered, and for the data source connection pool in the cache that exceeds the least usage, the data source connection pool is removed from the cache.

[0089] Removal message removal: After receiving the cache removal message, for the data source connection pool specified in the cache removal message, the data source connection pool is removed from the cache.

[0090] As a specific implementation, the data source connection pool in the cache is removed, including the following steps:

[0091] a, check if there is still an active connection in the data source connection pool to be removed being used, if yes, execute step b, otherwise execute step c;

[0092] b, remove the data source connection pool from the cache, wait for the active connection to complete the work, and when all connections in the data source connection pool are idle, close all connections in the data source connection pool and destroy the data source connection pool;

[0093] c, close all connections in the data source connection pool and destroy the data source connection pool, and remove the data source connection pool from the cache.

[0094] The method of the embodiment puts the data source connection pool into the cache, to avoid consuming memory, removes the cache according to the elimination strategy, listens to the changes of the data source connection information in real time, monitors the connection and use state of the data source, removes the old data source in the cache at the same time of the change of the data source information, improves the resource reuse, and reduces the resource overhead.

[0095] Embodiment 2:

[0096] The application discloses a multi-source data management system based on cache, which comprises a cache initialization module, a cache adding module and a cache removing module.

[0097] The cache initialization module is used for performing: configuring basic parameters of the cache to form a configuration file, reading the configuration file when a business system starts and initializing the local cache through the configuration file, the cache being used for storing data source connection pools and being configured with a listening service, and listening to the connection and use state of the data source connection pools through the cache listening service.

[0098] In the embodiment, the basic parameters of the cache include a cache initialization capacity, a cache maximum capacity, an elimination strategy and an expiration time, the above basic parameters are maintained in the configuration file, and the local cache is initialized by reading the configuration file when the application system starts, the cache being used for storing data source connection pool information.

[0099] The elimination strategy is that when a new data source connection pool is created but the capacity of the current cache reaches the capacity upper limit, the elimination strategy is triggered, the data source connection pool with the least use frequency in the current cache is removed from the cache, and the newly created data source connection pool is stored in the cache.

[0100] The expiration time is a predetermined period of time during which the data source connection pool in the cache is not used, and when the data source connection pool in the cache is not used for more than the predetermined period of time, the data source connection pool is regarded as an expired data source connection pool.

[0101] Meanwhile, the cache initialization module is configured to configure a listening service, which includes three services, i.e., a cache listening service, a cache invalidation listening service and a data source invalidation listening service.

[0102] The cache listening service is configured to listen to the number of data source connection pools in the cache, the time when the data source connection pool is added to the cache, the invalidation time, and the state of the data source connection pool.

[0103] The cache invalidation listening service is triggered when the data source connection pool is removed, so as to close the data source connection pool and avoid resource consumption caused by memory leakage.

[0104] The data source invalidation listening service is configured to determine whether there is a data source connection pool corresponding to the data source in the cache when the data source itself is deleted by the business system, and if so, the data source connection pool is removed from the cache.

[0105] The cache adding module is configured to execute the following operation: during the process of the business system processing data, for a newly created data source, the data source connection pool corresponding to the data source is put into the cache based on the data source information.

[0106] In the embodiment, the cache adding module is configured to execute the following operation to put the data source connection pool corresponding to the data source into the cache based on the data source information:

[0107] a. A user creates or maintains new data source connection information;

[0108] b. After the business system acquires the new data source connection information, the data source connection information is tested, if the test is successful, step c is executed, otherwise, step a is returned to modify the data source connection information again;

[0109] c. A unique identifier is generated based on the data source connection information through an MD5 algorithm, the unique identifier is configured to determine whether the data source connection pool exists in the cache, if not, step d is executed, if so, the cache adding is ended;

[0110] d. According to the data source information, the corresponding data source connection pool is created, and it is determined whether the capacity of the current cache reaches the maximum capacity of the cache, if so, an eviction policy is triggered, the data source connection pool with the least usage frequency in the cache is removed from the cache based on the eviction policy, and then the newly created data source connection pool is stored in the cache.

[0111] The data source connection information includes a data source connection address, a data source account, a database name and a database version number.

[0112] The cache removing module is configured to execute the following operation: the data source connection pool in the cache is removed.

[0113] In the embodiment, the data source connection pool in the cache is removed, including three cases of cache expiration removal, elimination strategy removal and removal message removal.

[0114] Cache expiration removal: when the unused time of the data source connection pool in the cache exceeds the expiration time, it is determined that the data source connection pool is expired, and the data source connection pool is removed from the cache for the expired data source connection pool.

[0115] Elimination strategy removal: when a new data source connection pool is created, if the capacity of the current cache reaches the maximum cache capacity, the elimination strategy is triggered, and the data source connection pool in the cache that exceeds the minimum usage is removed from the cache.

[0116] Removal message removal: after receiving the cache removal message, the data source connection pool specified in the cache removal message is removed from the cache.

[0117] As a specific implementation, the cache removal module is used to perform the following operations to remove the data source connection pool in the cache:

[0118] a. Check whether there is an active connection in the data source connection pool to be removed that is still being used, if yes, perform step b, otherwise perform step c;

[0119] b. Remove the data source connection pool from the cache, wait until all connections in the data source connection pool are in an idle state after the active connection completes the work, then close all connections in the data source connection pool and destroy the data source connection pool;

[0120] c. Close all connections in the data source connection pool and destroy the data source connection pool, and remove the data source connection pool from the cache.

[0121] The above has disclosed and illustrated the present application in detail through the accompanying drawings and preferred embodiments, however, the present application is not limited to these disclosed embodiments, and those skilled in the art can know that the code review means in different embodiments can be combined to obtain more embodiments of the present application, and these embodiments are also within the protection scope of the present application.

Claims

1. A cache-based multi-source data management method, characterized in that, The method for storing a data source connection pool in a cache comprises the following steps: cache initialization: configuring basic parameters of the cache to form a configuration file, reading the configuration file when the business system starts, and initializing the local cache through the configuration file, the cache being used to store a data source connection pool, and the cache being configured with a listening service to listen to the connection and use state of the data source connection pool; cache addition: during the process of the business system processing data, for a newly created data source, based on data source information, the data source connection pool corresponding to the data source is put into the cache; cache removal: removing the data source connection pool in the cache; wherein the basic parameters of the cache include cache initialization capacity, cache maximum capacity, elimination strategy, and expiration time; wherein the elimination strategy is that when a new data source connection pool is created but the current cache capacity reaches the capacity upper limit, the elimination strategy is triggered, for the data source connection pool with the least use frequency in the current cache, the data source connection pool is removed from the cache, and the newly created data source connection pool is stored in the cache; the expiration time is a predetermined period of time during which the data source connection pool in the cache is not used, when the data source connection pool in the cache is not used for more than the predetermined period of time, the data source connection pool is considered as an expired data source connection pool; removing the data source connection pool in the cache includes the following three cases: cache expiration removal: when the non-use time of the data source connection pool in the cache exceeds the expiration time, it is determined that the data source connection pool is expired, and the data source connection pool is removed from the cache; elimination strategy removal: when a new data source connection pool is created, if the current cache capacity reaches the maximum cache capacity, the elimination strategy is triggered, and for the data source connection pool with the least use frequency in the cache, the data source connection pool is removed from the cache; removal message removal: after receiving a cache removal message, for the data source connection pool specified in the cache removal message, the data source connection pool is removed from the cache.

2. The cache-based multi-source data management method according to claim 1, characterized in that, The listening service includes: cache listening, which is used to listen to the number of data source connection pools in the cache, the time when the data source connection pool is added to the cache, the invalidation time, and the state of the data source connection pool; cache invalidation listening, which is triggered when the data source connection pool is removed, and is used to close the data source connection pool; data source invalidation listening, which is used to determine whether there is a data source connection pool corresponding to the data source in the cache when the data source itself is deleted by the business system, and if so, the data source connection pool is removed from the cache.

3. The cache-based multi-source data management method of claim 1, wherein, For a newly created data source, based on data source information, the data source connection pool corresponding to the data source is put into the cache, including the following steps: a. A user creates or maintains new data source connection information; b. After the business system obtains the new data source connection information, the data source connection information is tested, if the test is successful, step c is executed, otherwise, the data source connection information is modified again in step a; c. generating a unique identifier based on the data source connection information by using MD5 algorithm, the unique identifier judging whether the data source connection pool exists in the cache, if not, executing step d, if yes, adding the cache and ending; d. creating the corresponding data source connection pool according to the data source information, and judging whether the current cache capacity reaches the maximum cache capacity, if yes, triggering the elimination strategy, removing the data source connection pool with the least usage frequency from the cache based on the elimination strategy, and storing the newly created data source connection pool into the cache; The data source connection information includes data source connection address, data source account, database name, and database version number.

4. The cache-based multi-source data management method of claim 1, wherein, The removal of the data source connection pool in the cache includes the following steps: a. checking whether there is still an active connection being used in the data source connection pool to be removed, if yes, executing step b, otherwise executing step c; b. removing the data source connection pool from the cache, waiting for the active connection to complete the work, and when all connections in the data source connection pool are in idle state, closing all connections in the data source connection pool and destroying the data source connection pool; c. closing all connections in the data source connection pool and destroying the data source connection pool, and removing the data source connection pool from the cache.

5. A cache-based multi-source data management system, characterized in that, The system for executing the cache-based multi-source data management method according to any one of claims 1-4 comprises: a cache initialization module, which is configured to execute: configuring basic parameters of the cache to form a configuration file, reading the configuration file when the business system starts, and initializing the local cache through the configuration file, the cache being used to store data source connection pools, and configuring a listening service for the cache to listen to the connection and usage state of the data source connection pools; a cache addition module, which is configured to execute: during the data processing of the business system, for a newly created data source, based on the data source information, storing the corresponding data source connection pool of the data source into the cache; a cache removal module, which is configured to execute: removing the data source connection pool in the cache; The basic parameters of the cache include cache initialization capacity, cache maximum capacity, elimination strategy, and expiration time. The elimination strategy is: when a new data source connection pool is created but the current cache capacity reaches the upper limit, the elimination strategy is triggered, the data source connection pool with the least usage frequency in the current cache is removed from the cache, and the newly created data source connection pool is stored in the cache. The expiration time is a predetermined period of time during which the data source connection pool in the cache is not used, and when the data source connection pool in the cache is not used for more than the predetermined period of time, the data source connection pool is considered as an expired data source connection pool. The removal of the data source connection pool in the cache includes the following three cases: Cache expiration removal: when the usage time of the data source connection pool in the cache exceeds the expiration time, the data source connection pool is determined to be expired, and the data source connection pool is removed from the cache; Eviction policy removal: when a new data source connection pool is created, if the current cache capacity reaches the maximum cache capacity, the eviction policy is triggered, and for the data source connection pool that exceeds the least frequently used in the cache, the data source connection pool is removed from the cache; Remove message removal: after receiving the cache removal message, for the data source connection pool specified in the cache removal message, the data source connection pool is removed from the cache.

6. The cache-based multi-source data management system of claim 5, wherein, The monitoring service includes: Cache monitoring, which is used to monitor the number of data source connection pools in the cache, the time when the data source connection pool is added to the cache and the invalidation time, and the state of the data source connection pool; Cache invalidation monitoring, which is triggered when the data source connection pool is removed, and is used to close the data source connection pool; Data source invalidation monitoring, which is used to determine whether there is a data source connection pool corresponding to the data source in the cache when the data source itself is deleted by the business system, and if so, the data source connection pool is removed from the cache.

7. The cache-based multi-source data management system of claim 5, wherein, The cache adding module is used to perform the following operations to put the data source connection pool corresponding to the data source into the cache based on the data source information: a. User creates or maintains new data source connection information; b. After the business system obtains the new data source connection information, the data source connection information is tested, and if the test is successful, step c is executed, otherwise, step a is returned to modify the data source connection information; c. Based on the data source connection information, a unique identifier is generated by MD5 algorithm, which determines whether the cache exists the data source connection pool, if not, step d is executed, if exists, the cache adding is ended; d. According to the data source information, the corresponding data source connection pool is created, and it is determined whether the current cache capacity reaches the maximum cache capacity, if yes, the eviction policy is triggered, and based on the eviction policy, the data source connection pool with the least usage frequency in the cache is removed from the cache, and the newly created data source connection pool is stored in the cache; Wherein, the data source connection information includes data source connection address, data source account, database name, and database version number.

8. The cache-based multi-source data management system of claim 5, wherein, The cache removal module is used to perform the following to remove the data source connection pool in the cache: a. Check whether there are active connections still being used in the data source connection pool to be removed, if yes, execute step b, otherwise execute step c; b. Remove the data source connection pool from the cache, wait for the active connection to complete the work, and when all connections in the data source connection pool are in idle state, close all connections in the data source connection pool and destroy the data source connection pool; c. Close all connections in the data source connection pool and destroy the data source connection pool, and remove the data source connection pool from the cache; Wherein, the removal of the data source connection pool in the cache includes the following three cases: Cache expiration removal: the data source connection pool in the cache that exceeds the predetermined period of use is regarded as an expired data source connection pool, and the data source connection pool is removed from the cache; Eviction policy triggers removal: when a new data source connection pool is created, but the current cache capacity reaches the capacity upper limit, triggers the eviction policy, removes the data source connection pool from the cache based on the eviction policy for the data source connection with the least usage in the current cache, and stores the newly created data source connection pool in the cache; Based on the message removal: based on the received cache removal message, remove the data source connection pool from the cache for the data source connection pool specified in the cache removal message.

Images