Database connection pool management method and device based on DataX

Abstract

The invention discloses a database connection pool management method and device based on DataX, and solves the problems of low resource utilization rate, concurrence disorder, thread safety and the like caused by lack of a built-in connection pool in the existing DataX. The method comprises the following steps: deploying an independent connection pool service (CPS), integrating a third-party connection pool and configuring a global maximum connection number; the DataX communicates with the CPS through a network, and each Channel independently applies for connection; the CPS dynamically adjusts the DataX parallelism degree (limitation or waiting when the DataX parallelism degree is insufficient) according to the available connection number, and allocates a unique connection identifier; the DataX sends an SQL request through the identifier, and the CPS calls the third-party connection pool for execution and returns a result; and after the task is completed, notifying the CPS to release the connection. Connection multiplexing is achieved through centralized management, the physical connection sharing risk is avoided, the parallelism degree is dynamically regulated and controlled to guarantee efficient utilization of resources, and DataX task stability and thread safety are improved.

Description

Technical Field

[0001] This application belongs to the field of database synchronization and data migration technology, and specifically relates to a database connection pool management method, apparatus, computer-readable storage medium, and electronic device based on DataX. Background Technology

[0002] With the rapid development of big data technology, data integration and synchronization have become crucial aspects of enterprise data governance. DataX, a widely adopted open-source data integration framework, supports efficient data transmission between heterogeneous data sources. DataX employs a Framework + Plugin architecture, using a Reader plugin to read source data and a Writer plugin to write target data, achieving flexible data synchronization capabilities.

[0003] In actual operation, DataX needs to continuously interact with both the source and target databases frequently to complete data extraction, transformation, and loading (ETL) operations. However, the current DataX framework itself does not have a built-in unified database connection pool management mechanism. This means that in DataX's default mode, each database operation may independently establish and close a database connection, leading to the following serious problems: First, there is resource waste and low execution efficiency. Due to the lack of connection reuse capabilities, DataX must independently complete the database connection establishment (TCP handshake, authentication negotiation, etc.) and release (connection recycling, resource destruction) process for each database operation (such as writing or querying a single data record). Such repetitive network interactions and system calls consume a large amount of computing resources and network bandwidth. Especially in scenarios involving massive small-batch data reads and writes (such as log synchronization, high-frequency incremental updates), the redundant overhead of connection management will significantly reduce the overall synchronization efficiency.

[0004] Secondly, there is a risk of uncontrolled connection resource management. When multiple DataX tasks are executed in parallel, each task independently requests and releases database connections, lacking a globally unified connection pool management component to schedule and limit connection usage by each task. This model can easily lead to a surge in the number of database connections in a short period of time, exceeding the maximum connection threshold of the database instance, which can then cause problems such as connection refusal or slow query backlog, ultimately affecting the overall stability of the database and data integration system.

[0005] Thirdly, there are issues with thread safety and data consistency. DataX internally uses multiple Channels (data transfer sub-threads) to achieve parallel data processing. If different Channels attempt to share the same database connection, thread safety issues will arise due to multiple threads concurrently operating on the same connection object (such as sharing a Statement or ResultSet). Typical manifestations include data write errors (such as incorrect row offsets), transaction context conflicts, or abnormal connection states. In severe cases, this can lead to task failure or database corruption.

[0006] Fourth, the system's scalability is limited. The static allocation model of database connection resources is difficult to adapt to dynamically changing load scenarios. When the database needs to adjust the number of available connections due to performance fluctuations or resource limitations, DataX cannot dynamically scale up or down the number of parallel processing threads or adjust the connection usage strategy based on real-time load. Similarly, when the scale of data synchronization tasks expands, the bottleneck of connection resources will directly restrict the throughput improvement of DataX, and it cannot achieve elastic matching between computing resources and database connection resources.

[0007] In summary, the existing DataX framework suffers from inefficiencies in database connection management, including inefficiency, security, and scalability. There is an urgent need for a unified, efficient, stable, and secure database connection pool management solution. Summary of the Invention

[0008] To address the aforementioned issues, this application proposes a novel database connection pool management method based on DataX. This invention aims to solve the problems of low resource utilization, chaotic connection management during multi-task concurrency, lack of thread safety guarantees, and thread safety and connection sharing between DataX Channels caused by the lack of a built-in database connection pool management mechanism in the existing DataX framework, providing an efficient, stable, and controllable database connection pool management solution.

[0009] To achieve the above objectives, the present invention employs the following technical strategies: (1) Introducing an independent distributed database connection pool service (CPS): By introducing an independent and deployable "database connection pool service (CPS)," the management capability of the connection pool is separated from the DataX job process, realizing centralized and global management of the connection pool, and decoupling the life cycle of CPS from the DataX task process.

[0010] (2) Connection management based on network communication: DataX communicates with CPS through standard network protocols (such as HTTP / REST) ​​to realize the application, allocation, execution and release of connections. This approach has high scalability and supports service deployment across machines and networks.

[0011] (3) Role change of DataX: DataX only acts as the applicant and user of the connection, and does not directly manage the physical connection. This greatly simplifies the internal implementation of DataX, while encapsulating the complex management logic of the connection pool in CPS.

[0012] (4) Independent Channel Requests and Global Control of CPS: Each Channel of DataX independently initiates a connection request to CPS. CPS is responsible for implementing the global maximum number of connections limit, connection reuse, and a unified waiting and scheduling mechanism when there are insufficient connections.

[0013] (5) Dynamic parallelism negotiation mechanism: When DataX starts up, it communicates with CPS and dynamically adjusts its own parallelism (number of channels) according to the actual number of available connections fed back by CPS, so as to ensure that the task can still start and run efficiently and stably under resource constraints.

[0014] (6) Thread safety guarantee under network isolation: The Channel inside DataX obtains the connection identifier from CPS through independent network requests. CPS is responsible for the isolation and management of physical connections, thereby avoiding the thread safety issues that may occur when multiple threads of DataX share the same physical database connection.

[0015] The design architecture, core functions, and execution flow of this invention are described below: 1. Integration of DataX with third-party connection pools: Introduce a separate, configurable connection pool service (CPS).

[0016] DataX no longer directly manages database connections when performing data synchronization tasks.

[0017] DataX sends SQL statements, data, and all the information needed to execute the task (such as database type, connection parameters, SQL statements, data to be processed, etc.) to CPS through predefined interfaces (such as RESTful API or message queues).

[0018] After receiving a task request from DataX, CPS executes SQL operations within its own JVM process using its built-in third-party database connection pools (such as HikariCP, Druid, c3p0, etc.).

[0019] CPS does not directly expose the physical database connection object (Connection) to DataX.

[0020] CPS serializes the results of SQL execution (including data, status codes, error messages, etc.) and returns them to DataX through a predefined interface.

[0021] 2. Using connections in DataX (asking for and returning connections): When each job in DataX starts, it establishes a communication channel with CPS.

[0022] When a DataX Channel (sub-thread) needs to access the database, the Channel does not directly request a connection to the database. Instead, the Channel sends a "connection request" to the CPS, carrying the information required to execute the SQL.

[0023] CPS obtains an available physical connection from its connection pool based on the availability of connections in the pool.

[0024] After obtaining a physical connection, CPS executes the SQL statements passed from the DataX Channel on that physical connection.

[0025] After the SQL is executed, CPS returns the result to the DataX Channel.

[0026] The key to this design is that CPS is responsible for managing the lifecycle of physical connections. DataX Channel does not need to explicitly "return" connections; instead, it notifies CPS when a task is completed or the channel is closed. Internally, CPS automatically manages the reuse and release of connections based on the connection pool configuration.

[0027] 3. DataX Parallel Number Control: CPS maintains a global database connection pool and can monitor in real time the number of available connections (available_connections), the current total number of connections (total_connections), and the configured maximum number of connections (max_connections).

[0028] When DataX submits a new task, CPS will determine the number of channels (requested_channels) configured in the task and the current available_connections.

[0029] The judgment principles are as follows: If available_connections >= requested_channels: CPS allows DataX to start sub-task channels according to the number of requested_channels configured by it, and CPS will reserve enough connections for these channels.

[0030] If available_connections < requested_channels: CPS limits the actual number of available channels to available_connections. That is to say, the parallelism of the DataX task (the number of sub-task channels) will be adjusted to available_connections. This design can prevent some channels from failing to start or being blocked due to insufficient connections.

[0031] Connection pool expansion and waiting: When available_connections < requested_channels but total_connections < max_connections, CPS can dynamically create new physical database connections as needed and add them to the connection pool until the requests of DataX channels are satisfied or max_connections is reached.

[0032] When total_connections reaches max_connections and available_connections < requested_channels, the DataX task (or some channels) will enter a waiting state until new available connections are released from the connection pool.

[0033] Specifically, the present application provides the following technical solutions: The first aspect of the present application provides a database connection pool management method based on DataX, as Figure 4 shown. This method includes the following steps: S1. Deploy CPS (Connection Pool Service) independent of the DataX task process. The CPS is configured with a global maximum connection number limit and integrates at least one third-party database connection pool; S2. After the DataX task is started, establish a connection with the CPS through the network communication interface; S3. For each Channel (data transfer sub-thread) in the DataX task, send a connection application request to the CPS; S4. The CPS determines whether to allocate physical connections or adjust the parallelism of the DataX task based on the current number of available connections and the global maximum number of connections: If the number of available connections is greater than or equal to the number of requests, assign a unique connection identifier to each Channel and provide feedback. If the number of available connections is less than the number of requests, the actual number of available connections will be used as the actual parallelism of the DataX task, and the adjusted parallelism will be fed back. S5. The DataX task sends an SQL execution request to the CPS through a connection identifier. The CPS calls a third-party database connection pool based on the connection identifier to execute the SQL operation and return the result. S6. After the DataX task completes the database operation, it notifies the CPS to release the physical connection with the corresponding connection identifier. The CPS then returns the physical connection to the third-party database connection pool and updates the connection status.

[0034] Furthermore, in the method of this application, the CPS in step S1 maintains the global connection count status, including the current total number of connections, the number of available connections, and the global maximum number of connections, and limits the total number of connections for the DataX task through the global maximum number of connections.

[0035] Furthermore, in the method of this application, in step S4, the CPS dynamically creates new physical connections to the third-party database connection pool when the number of available connections is less than the number of requests and the current total number of connections is less than the global maximum number of connections, until the number of requests is met or the global maximum number of connections is reached.

[0036] Furthermore, in the method of this application, in step S4, when the number of available connections is less than the number of requests and the current total number of connections equals the global maximum number of connections, the Channel of the DataX task enters a waiting state and periodically retryes connection requests until the CPS releases available connections, wakes up, and allocates them.

[0037] Furthermore, in the method of this application, the network communication interface uses a standard network protocol (such as HTTP / REST or message queue) to realize the interaction between DataX and CPS.

[0038] Furthermore, in the method of this application, in step S3, each Channel independently sends a connection request to the CPS, and the CPS assigns an independent connection identifier to each Channel to ensure physical connection isolation between different Channels.

[0039] Furthermore, in the method of this application, the CPS described in step S5 does not expose the physical connection object to DataX, but only completes the routing of SQL execution requests and the return of results through the connection identifier.

[0040] A second aspect of this application provides a database connection pool management device based on DataX, the device comprising: The CPS deployment module is used to deploy a CPS (Connection Pool Service) independent of the DataX task process. The CPS is configured with a global maximum connection limit and integrates at least one third-party database connection pool. The connection module is used to establish a connection with CPS via a network communication interface after the DataX task starts; The request module is used to send connection request requests to CPS for each Channel (data transfer sub-thread) in the DataX task; The parallelism management module is used by CPS to determine whether to allocate physical connections or adjust the parallelism of DataX tasks based on the current number of available connections and the global maximum number of connections. If the number of available connections is greater than or equal to the number of requests, assign a unique connection identifier to each Channel and provide feedback. If the number of available connections is less than the number of requests, the actual number of available connections will be used as the actual parallelism of the DataX task, and the adjusted parallelism will be fed back. The execution module is used by DataX tasks to send SQL execution requests to CPS via connection identifiers. CPS then calls a third-party database connection pool based on the connection identifiers to execute SQL operations and return the results. The return module is used to notify CPS to release the physical connection with the corresponding connection identifier after the DataX task completes the database operation. CPS then returns the physical connection to the third-party database connection pool and updates the connection status.

[0041] The device implements the steps of the aforementioned DataX-based database connection pool management method during operation.

[0042] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned DataX-based database connection pool management method.

[0043] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned DataX-based database connection pool management method.

[0044] In summary, compared with the prior art, the solution of the present invention has the following advantages: (1) Distributed centralized connection pool management capability: By decoupling the connection pool management function to the independent component CPS, global connection resource scheduling across multiple DataX task instances is realized, significantly improving the utilization rate of database connection resources and global control capabilities.

[0045] (2) Decoupling the DataX process from the connection pool lifecycle: CPS runs independently of the DataX task process. Its lifecycle management is not limited to a single DataX task. It supports flexible start-up and shutdown and independent maintenance. The start-up or termination of the DataX task will not affect the continuous availability of the connection pool.

[0046] (3) Synchronization performance optimization: Relying on the database connection reuse mechanism, the network interaction and system call overhead during the connection establishment and release process are effectively reduced, which directly improves the data synchronization execution efficiency of DataX tasks.

[0047] (4) Enhanced database stability: By implementing global maximum connection limit and dynamic parallelism adjustment strategy through CPS, database overload caused by connection resource exhaustion is avoided, ensuring the stable operation of database services.

[0048] (5) Internal thread safety guarantee: Each Channel is forced to independently request a connection from the CPS, and the connection isolation management is implemented by the CPS level, which completely eliminates the thread safety risks (such as data corruption, abnormal connection status, etc.) caused by multiple threads sharing the connection inside DataX.

[0049] (6) Enhanced availability and scalability: Independently deployed CPS supports convenient monitoring, maintenance and elastic expansion (such as clustered deployment), effectively enhancing the overall high availability and reliability of the data integration system.

[0050] (7) Separation of duties and ease of maintenance: DataX focuses on the implementation of core logic for data integration, while CPS focuses on connection pool management. The responsibilities of the two are clearly defined, reducing system coupling and improving development and maintenance efficiency.

[0051] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of this application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0053] Figure 1 This is a diagram of the overall design architecture of the present invention.

[0054] Figure 2 This is a flowchart of the parallel number control process for DataX in the present invention.

[0055] Figure 3 This is a flowchart of the execution process of each Channel in DataX in the present invention.

[0056] Figure 4 This is a flowchart illustrating the overall implementation process of the database connection pool management method based on DataX in this application.

[0057] Figure 5 This is a structural diagram of the database connection pool management device based on DataX in this application.

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0060] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0061] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0062] Example: A Database Connection Pool Management Method Based on DataX 1. System Architecture Figure 1 The diagram shows the overall design architecture of the present invention, wherein: DataX: As the "submitter" of tasks, it is responsible for data preparation, SQL construction, and task scheduling.

[0063] Connection Pool Service (CPS): A standalone backend service that integrates one or more third-party database connection pools. CPS is responsible for receiving task requests from DataX, managing database connections, executing SQL, and returning results.

[0064] Third-party database connection pools: hosted internally by CPS, such as HikariCP, Druid, c3p0, etc.

[0065] 2. Independent connection pool service Independent Connection Pool Service: Deploy a standalone "Database Connection Pool Service" (CPS) that is responsible for instantiating, configuring, and managing one or more third-party database connection pools. CPS runs as an independent process or service and has its own lifecycle.

[0066] Network communication interface: CPS provides a network interface (such as HTTP / REST API) to receive connection requests, execute SQL requests and release connection requests from DataX tasks.

[0067] Global connection limit control: CPS reads the maximum database connection limit (maxGlobalConnections) from the global configuration. All management logic within CPS follows this global limit, ensuring that the total number of connections requested through CPS does not exceed this limit.

[0068] Third-party connection pool integration: CPS internally uses an open-source third-party database connection pool library to convert DataX task configurations (database URL, user, password, driver class, etc.) into configuration parameters of the third-party connection pool, and to create and manage underlying connection pool instances.

[0069] 3. Using connections in DataX (asking for and returning connections) Figure 3 The diagram shows the execution flow of each Channel in DataX in the present invention.

[0070] The role of DataX: The DataX task process no longer directly creates database connections, but acts as the "requester" and "user" of connections.

[0071] Network requests: DataX tasks encapsulate database connection information and requests (such as connection requests) into network messages through their internal adapter layer and send them to a separate CPS.

[0072] Connection request: (1) Each Channel (sub-thread) of the DataX task sends a "connection request" to CPS independently when a database connection is needed.

[0073] (2) After receiving the request, CPS determines whether it can provide a connection immediately based on its internal connection pool management logic.

[0074] (3) If CPS can allocate a connection, it will return a unique “connectionId”, which the DataX task will associate with the current Channel.

[0075] (4) If CPS cannot provide a connection immediately (for example, the number of connections has reached the limit), it will return a "wait" signal, and the DataX Channel thread will enter a waiting state and periodically retry the connection request until it succeeds.

[0076] Connection usage: DataX tasks, through the adaptation layer, use the obtained "connection identifier" to send requests such as SQL execution and data query to CPS. CPS finds the corresponding physical database connection based on the "connection identifier," executes the operation, and returns the result to DataX.

[0077] Connection return: After completing the database operation, the DataX task's Channel notifies the CPS to release the corresponding connection through the adaptation layer. Upon receiving the release request, the CPS returns the physical connection to the connection pool it manages and updates the connection status.

[0078] 4. DataX Parallel Number Control Figure 2 The diagram shows the parallel data control flow of DataX in the present invention.

[0079] Dynamic parallelism adjustment: When a DataX task starts, it will send an initial "connection request" to CPS based on the configured number of channels (for each expected channel request).

[0080] CPS reports the number of available connections: When responding to these initial requests, CPS reports back to DataX the actual number of database connections available in its currently managed connection pool (not the total number of connections).

[0081] DataX's parallelism decision: (1) The parallelism control module of DataX receives the number of available database connections from the CPS.

[0082] (2) If the number of available database connections reported by CPS is less than the number of channels configured by DataX, DataX will use the number of available connections as the actual number of channels and start the parallel processing of the subtask.

[0083] (3) DataX will use the number of channels configured as the number of parallel subtasks only when the number of available database connections reported by CPS is greater than or equal to the number of channels configured by DataX.

[0084] The above mechanism ensures that the parallelism of DataX tasks is always limited by the available resources in the connection pool, avoiding startup failures or performance bottlenecks caused by insufficient connections.

[0085] 5. Connection pool connection count control and waiting Maximum Global Connections (maxGlobalConnections): CPS controls the maximum number of connections across all third-party connection pools it manages through configuration parameters.

[0086] (1) Connection creation and allocation: When CPS receives a connection request from DataX and the current total number of connections is less than maxGlobalConnections, CPS will attempt to obtain a new connection from the third-party connection pool it manages.

[0087] If successful, CPS will bind the physical connection to the returned "connection identifier" and increment the total connection count.

[0088] (2) Waiting when the maximum number of connections is reached: When CPS detects that the current total number of connections has reached maxGlobalConnections and there are no available connections, new connection requests will be placed in the waiting queue inside CPS.

[0089] CPS will return a "wait" signal to DataX, and DataX's Channel thread will pause and retry periodically.

[0090] When a connection is returned, CPS will wake up the requests in the waiting queue and allocate a connection to them.

[0091] 6. Secure connections between DataX Channels Independent application mechanism: DataX's adaptation layer independently initiates a database connection application for each Channel (sub-thread) and obtains a unique connectionId from CPS.

[0092] Isolation and Avoidance of Sharing: DataX internally does not pass the connectionId or its corresponding physical connection information obtained by one Channel to other Channels. Each Channel maintains its own independent connectionId.

[0093] Network communication security: DataX interacts with CPS over the network. CPS manages the connection based on the connectionId, ensuring that database operations of different channels are isolated at the physical connection level, thus avoiding thread safety issues that may arise from directly sharing the same Connection object.

[0094] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0095] This example demonstrates how DataX interacts with a standalone connection pool service (CPS) developed in Java (Spring Boot) via an HTTP / REST API.

[0096] 1. Implementation of Independent Connection Pool Service (CPS): CPS Development: (1) Create a Spring Boot project as a CPS.

[0097] (2) ConnectionPoolService class (core): (a) For each database, maintain a global configuration maxGlobalConnections.

[0098] (b)Maintain a set Map<String, ConnectionInfo> to store available database connections, where the key is the connectionId (connection identifier, identified by UUID), and the value is the ConnectionInfo object, which contains the physical java.sql.Connection object and the database configuration (information such as URL, user, etc.) associated with the connection.

[0099] (c)Maintain a set Map<String, ChannelRequest> to store DataXChannel requests (channelId, dbConfig, etc.) waiting for connections, where the key is the channelId, and the value is the ChannelRequest object, which contains information such as channelId and dbConfig. Here, the channelId is the Channel identifier of the corresponding DataX task, and the dbConfig is the encapsulated database configuration information object.

[0100] (d)Introduce HikariCP as the underlying third-party connection pool. The CPS can dynamically create HikariDataSource instances for different database configurations according to the task requests of DataX.

[0101] (e)Request connection method requestConnection(dbConfig, channelId): Check if there is a matching and idle physical connection in the ConnectionInfo Map.

[0102] If not, and connectionInfo.size() < maxGlobalConnections, it means that the number of currently created available database connections has not reached the global maximum connection number for this database: 1) Try to obtain a new connection from the underlying HikariCP pool (find or create according to dbConfig).

[0103] 2) If successful, generate a unique connectionId.

[0104] 3) Store the new connection, connectionId, and dbConfig into the ConnectionInfo Map collection.

[0105] If connectionInfo.size() = maxGlobalConnections, it means that the number of available database connections created has reached the global maximum number of connections for that database, and subsequent connection requests will have to wait.

[0106] (f) The `releaseConnection(connectionId)` method returns the connection: Find the corresponding ConnectionInfo based on the connectionId.

[0107] Return the physical connection to the HikariCP connection pool.

[0108] Remove the entry from the ConnectionInfo Map collection.

[0109] Check the waiting queue. If there is a waiting request, remove it, attempt to allocate a connection to it, and return the newly allocated connectionId and availableConnections to DataX.

[0110] (g) Execute the SQL method executeSql(connectionId, sql, params): Find the physical connection based on the connectionId.

[0111] Execute the SQL and return the results.

[0112] (3) HTTP / REST Controller: Implements API interfaces such as / connect, / release, / execute, and / status for DataX to call.

[0113] Deployment: CPS runs independently as a microservice.

[0114] 2. DataX client adaptation layer and integration: RemoteDataSourceAccessor class (internal to DataX): (1) Initialization method init(): Reads connectionPoolServiceUrl from Job configuration and initializes HTTP client.

[0115] (2) Get connection method getConnection(channelId, dbConfig): (a) When the DataX job starts, it sends an HTTP POST request to the / connect interface of CPS to get the number of available connections for the corresponding database.

[0116] (b) When the number of available connections is greater than the currently set number of Channels, DataX starts a corresponding number of subtasks for parallel processing according to the set number of Channels. Each subtask obtains its own database connection from the connection pool.

[0117] (c) When the number of available connections is less than the number of Channels currently set, DataX will start a corresponding number of subtasks for parallel processing according to the number of available connections. Each subtask will obtain its own database connection from the connection pool.

[0118] (3) Release Connection(channelId): Get the connectionId corresponding to the channelId.

[0119] Send an HTTP POST request to the / release interface of CPS, carrying the connectionId.

[0120] (4) Execute the SQL method executeSql(channelId, sql, params): Get the connectionId corresponding to the channelId.

[0121] Send an HTTP POST request to the / execute interface of CPS, carrying connectionId, sql, and params.

[0122] Process the execution results returned by CPS.

[0123] Figure 5 The image shows a database connection pool management device based on DataX proposed in this application. The device includes: The CPS deployment module is used to deploy a CPS (Connection Pool Service) independent of the DataX task process. The CPS is configured with a global maximum connection limit and integrates at least one third-party database connection pool. The connection module is used to establish a connection with CPS via a network communication interface after the DataX task starts; The request module is used to send connection request requests to CPS for each Channel (data transfer sub-thread) in the DataX task; The parallelism management module is used by CPS to determine whether to allocate physical connections or adjust the parallelism of DataX tasks based on the current number of available connections and the global maximum number of connections. If the number of available connections is greater than or equal to the number of requests, assign a unique connection identifier to each Channel and provide feedback. If the number of available connections is less than the number of requests, the actual number of available connections will be used as the actual parallelism of the DataX task, and the adjusted parallelism will be fed back. The execution module is used by DataX tasks to send SQL execution requests to CPS via connection identifiers. CPS then calls a third-party database connection pool based on the connection identifiers to execute SQL operations and return the results. The return module is used to notify CPS to release the physical connection with the corresponding connection identifier after the DataX task completes the database operation. CPS then returns the physical connection to the third-party database connection pool and updates the connection status.

[0124] When the above-mentioned device is in operation, it implements the steps of the database connection pool management method based on DataX disclosed in this application.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. Each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0126] like Figure 6 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the DataX-based database connection pool management method described above.

[0127] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0128] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.

[0129] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.

[0130] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0131] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the database connection pool management method based on DataX disclosed in this application.

[0132] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0133] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0134] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0135] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0136] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0137] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0138] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0139] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0140] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0141] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0142] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0143] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the DataX-based database connection pool management method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0144] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0145] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A database connection pool management method based on DataX, characterized in that, The method includes: S1. Deploy a CPS independent of the DataX task process. The CPS is configured with a global maximum connection limit and integrates at least one third-party database connection pool. After the S2 and DataX tasks are started, they establish a connection with the CPS through the network communication interface; S3. For each Channel in the DataX task, send a connection request to the CPS; S4. The CPS determines whether to allocate physical connections or adjust the parallelism of the DataX task based on the current number of available connections and the global maximum number of connections. If the number of available connections is greater than or equal to the number of requests, assign a unique connection identifier to each Channel and provide feedback. If the number of available connections is less than the number of requests, the actual number of available connections will be used as the actual parallelism of the DataX task, and the adjusted parallelism will be fed back. S5. The DataX task sends an SQL execution request to the CPS through a connection identifier. The CPS calls a third-party database connection pool based on the connection identifier to execute the SQL operation and return the result. S6. After the DataX task completes the database operation, it notifies the CPS to release the physical connection with the corresponding connection identifier. The CPS then returns the physical connection to the third-party database connection pool and updates the connection status.

2. The method according to claim 1, characterized in that, In step S1, the CPS maintains the global connection count status, including the current total number of connections, the number of available connections, and the global maximum number of connections, and limits the total number of connections for the DataX task through the global maximum number of connections.

3. The method according to claim 1, characterized in that, In step S4, CPS dynamically creates new physical connections to the third-party database connection pool when the number of available connections is less than the number of requests and the current total number of connections is less than the global maximum number of connections, until the number of requests is met or the global maximum number of connections is reached.

4. The method according to claim 1, characterized in that, In step S4, when the number of available connections is less than the number of requests and the current total number of connections equals the global maximum number of connections, the Channel of the DataX task enters a waiting state and periodically retryes connection requests until the CPS releases available connections, wakes up, and allocates them.

5. The method according to claim 1, characterized in that, The network communication interface uses a standard network protocol to enable interaction between DataX and CPS.

6. The method according to claim 1, characterized in that, In step S3, each Channel independently sends a connection request to the CPS, and the CPS assigns an independent connection identifier to each Channel to ensure physical connection isolation between different Channels.

7. The method according to claim 1, characterized in that, In step S5, the CPS does not expose the physical connection object to DataX, but only completes the routing of SQL execution requests and the return of results through the connection identifier.

8. A database connection pool management device based on DataX, characterized in that, The device includes: The CPS deployment module is used to deploy CPS independent of the DataX task process. The CPS is configured with a global maximum connection limit and integrates at least one third-party database connection pool. The connection module is used to establish a connection with CPS via a network communication interface after the DataX task starts; The request module is used to send a connection request to the CPS for each Channel in the DataX task; The parallelism management module is used by CPS to determine whether to allocate physical connections or adjust the parallelism of DataX tasks based on the current number of available connections and the global maximum number of connections. If the number of available connections is greater than or equal to the number of requests, assign a unique connection identifier to each Channel and provide feedback. If the number of available connections is less than the number of requests, the actual number of available connections will be used as the actual parallelism of the DataX task, and the adjusted parallelism will be fed back. The execution module is used by DataX tasks to send SQL execution requests to CPS via connection identifiers. CPS then calls a third-party database connection pool based on the connection identifiers to execute SQL operations and return the results. The return module is used to notify CPS to release the physical connection with the corresponding connection identifier after the DataX task completes the database operation. CPS then returns the physical connection to the third-party database connection pool and updates the connection status.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the database connection pool management method based on DataX as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the DataX-based database connection pool management method as described in any one of claims 1-7.

Images