Data source access method and computing device

Abstract

The invention discloses a data source access method and computing equipment, and relates to the technical field of computing equipment. The method is applied to a first computing device. The method comprises the steps that a first task is received, the first task is used for indicating access to a first data source, the first task comprises a type item of the first data source, and the type item of the first data source comprises identification information of the first data source; determining identification information of the first data source based on the first task; acquiring connection information of the first data source from the second computing device based on the identification information of the first data source; based on the connection information of the first data source, obtaining a data source connection of the first data source, the data source connection of the first data source being used for representing a communication channel between the first computing device and the first data source; and accessing the first data source based on the data source connection of the first data source. The data source access flexibility is improved, and multiple types of data sources can be quickly accessed.

Description

Technical Field

[0001] This application relates to the field of computing device technology, and in particular to a data source access method and computing device. Background Technology

[0002] As digital transformation deepens across industries, various types of databases are constantly evolving, and enterprises are using increasingly diverse data sources, expanding from traditional relational and non-relational databases to various cloud storage and real-time streaming data sources. To support digital operations, business analysis, and decision-making, enterprises need to efficiently integrate these heterogeneous data sources. Although scheduling systems have built-in basic data source access capabilities, in practical applications they often face the problem of not being able to directly access some special or new data sources, affecting the efficiency and timeliness of enterprise data processing.

[0003] Therefore, how to achieve flexible adaptation and efficient access to various data sources is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a data source access method and computing device to achieve flexible adaptation and efficient access to various data sources.

[0005] In a first aspect, embodiments of this application provide a method for accessing a data source, applied to a first computing device. The method includes: receiving a first task, the first task indicating access to a first data source, the first task including a type item of the first data source, the type item including identification information of the first data source, the identification information of the first data source being used by the first computing device to obtain connection information of the first data source from a second computing device; obtaining the connection information of the first data source from the second computing device based on the identification information of the first data source; obtaining a data source connection of the first data source based on the connection information of the first data source, the data source connection of the first data source representing a communication channel between the first computing device and the first data source; and accessing the first data source based on the data source connection of the first data source.

[0006] Thus, based on the type item of the first data source included in the first task, the identification information of the first data source is obtained, and the connection information is obtained from the second computing device based on the identification information to create a data source connection for the first data source. There is no need to develop adaptation logic separately for each data source or manually configure the connection information. This simplifies the operation process of accessing the first data source, flexibly adapts to and efficiently accesses various data sources, and improves the flexibility and efficiency of data source access.

[0007] In one possible implementation, the connection information of the first data source includes the address of the target server where the first data source is located, the port number of the port on the target server through which the first data source provides services, and the account and password for accessing the first data source; based on the connection information of the first data source, obtaining the data source connection of the first data source includes: obtaining a first data source instance based on the connection information of the first data source, the first data source instance being used to obtain the data source connection of the first data source; and calling the first data source instance to obtain the data source connection of the first data source.

[0008] Thus, by first obtaining the first data source instance, since the data source instance has the function of obtaining the data source connection, the data source connection of the first data source can be obtained.

[0009] In one possible implementation, obtaining a first data source instance based on the connection information of the first data source includes: obtaining a target object and configuring the connection information of the first data source into the target object, the target object being used to encapsulate the connection information of the first data source; calling a first method and using the target object configured with the connection information of the first data source as an input parameter of the first method to create a first data source instance, the first method being used to create the first data source instance; and calling the first data source instance to obtain the data source connection of the first data source, including: calling the first data source instance to create the data source connection of the first data source.

[0010] This approach is applicable to scenarios where a first data source instance is created for the first time based on the connection information of the first data source. It enables the entire process of configuring the target object, creating the first data source instance, and generating the data source connection to the first data source to be completed during the initial connection process, improving the standardization, reliability, and maintainability of the first data source connection.

[0011] In one possible implementation, obtaining a first data source instance based on the connection information of a first data source includes: determining, based on the connection information of the first data source, a data source instance corresponding to the connection information of the first data source from a set of data source instances, serving as the first data source instance; the set of data source instances includes data source instances corresponding to connection information of different data sources; the first data source instance includes multiple data source connections, the multiple data source connections having different connection parameters, the connection parameters including one or more of connection quantity, connection status, and lifecycle; and invoking the first data source instance to obtain the data source connection of the first data source includes: invoking the first data source instance to determine the data source connection of the first data source from the multiple data source connections based on the connection parameters.

[0012] In this way, by accurately matching the first data source instance from the data source instance set and obtaining the data source connection of the first data source based on the connection parameters, we can achieve efficient reuse of data source connections and control of resource consumption, while ensuring the availability of data source connections and improving the efficiency, stability and resource utilization of obtaining the first data source connection.

[0013] In one possible implementation, before accessing the first data source based on the data source connection to the first data source, the method further includes: calling a second method to verify whether the data source connection to the first data source is valid, the second method being used to verify whether the data source connection is valid; accessing the first data source based on the data source connection to the first data source includes: if the verification result of the data source connection to the first data source is valid, accessing the first data source based on the data source connection to the first data source.

[0014] In this way, by first verifying the validity of the first data source connection and then accessing the first data source based on the valid first data source connection, the access failure problem caused by the invalid first data source connection can be avoided, thereby improving the success rate and stability of the first data source access and ensuring the reliability of the data interaction process.

[0015] In one possible implementation, before receiving the first task, the method further includes: responding to the user's selection of a type item for the first data source, displaying a configuration interface, the configuration interface including a first option for the user to configure the identification information of the first data source; and receiving the identification information of the first data source configured by the user in the first option.

[0016] In this way, by developing the native data source management module of the extended scheduling system, the identification information of the first data source can be configured. This not only enables the addition of the first data source type item, but also dynamically renders the matching configuration interface when the user selects the type. It can also accurately receive the identification information configured by the user, providing key input for obtaining connection information from the second device based on the identification information of the first data source. This enables the scheduling system to seamlessly extend to heterogeneous data sources, improving the flexibility of the scheduling system and the ease of use for users.

[0017] In one possible implementation, before obtaining the connection information of the first data source from the second computing device based on the identification information of the first data source, the method further includes: obtaining the interface configuration information of the second computing device from a preset configuration file, the interface configuration information including the interface address and interaction method of the second computing device; obtaining the connection information of the first data source from the second computing device based on the identification information of the first data source, including: obtaining the connection information of the first data source from the second computing device based on the identification information of the first data source and the interface configuration information of the second computing device.

[0018] In this way, by first obtaining the interface configuration information of the second computing device from the preset configuration file, and then combining it with the identification information of the first data source to obtain the connection information from the second computing device, unified management and flexible adjustment of the interface configuration can be achieved, while ensuring the standardization and accuracy of the first data source connection information acquisition, thereby improving the reliability and maintainability of the data acquisition process.

[0019] In one possible implementation, the method further includes: receiving a second task, the second task indicating access to a second data source, the second task including a type item of the second data source, the type item of the second data source including connection information of the second data source; obtaining a data source connection of the second data source based on the connection information of the second data source, the data source connection of the second data source representing a communication channel between the first computing device and the second data source; and accessing the second data source based on the data source connection of the second data source.

[0020] Thus, based on the connection information of the second data source, the data source connection of the second data source is obtained, and the second data source is accessed, achieving targeted and reliable access to the second data source.

[0021] In one possible implementation, the method further includes: disconnecting the data source connection of the first data source in response to the completion of the first task.

[0022] This avoids the primary data source connection from occupying system resources and network links for extended periods, and reduces the consumption of idle data source connections in the connection pool.

[0023] Secondly, embodiments of this application provide a computing device, including: a memory and a processor. The memory is used to store program instructions. The processor is used to execute the program instructions, causing the computing device to perform a data source access method as described in the first aspect or any possible implementation thereof.

[0024] Thirdly, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed on a computing device, cause the computing device to perform a data source access method as described in the first aspect or any possible implementation thereof.

[0025] Fourthly, a computer program product is provided, the computer program product including computer execution instructions, which, when executed on a computing device, cause the computing device to perform a data source access method as described in the first aspect or any possible implementation of the first aspect.

[0026] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the first aspect and the technical effects of different implementation methods in the first aspect, which will not be repeated here.

[0027] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0028] Figure 1 A schematic diagram of the structure of an ERP, an ERP data source, and a scheduling system provided for embodiments of this application; Figure 2 This application provides a schematic diagram of the interaction between a first computing device and a second computing device. Figure 3 A schematic diagram of the structure of a data development module and a scheduling system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application; Figure 5 A schematic diagram of a process for configuring a first data source is provided in an embodiment of this application; Figure 6 A schematic diagram of a visual interactive interface for a data source management module provided in an embodiment of this application; Figure 7 A schematic diagram illustrating a process for deploying a data source plugin, provided as an embodiment of this application; Figure 8 A flowchart illustrating the configuration of a type item of a first data source in a scheduling system, provided as an embodiment of this application; Figure 9 This application provides a schematic diagram of a process for accessing a first data source according to an embodiment of the present application; Figure 10 A schematic diagram of an interactive interface provided by a data development module according to an embodiment of this application; Figure 11 A schematic diagram illustrating another process for accessing the first data source provided in an embodiment of this application; Figure 12 A schematic diagram of a visualization interface for a scheduling system provided in an embodiment of this application; Figure 13 This is a visual schematic diagram of a data source access provided in an embodiment of this application; Figure 14 This is a schematic diagram of another computing device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0030] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0031] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0032] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0033] The following provides an exemplary description of the application scenarios of the embodiments of this application.

[0034] With the deepening of digital transformation in various industries, various types of databases are constantly evolving, and the data sources used by enterprises are becoming increasingly rich and diverse, such as expanding from traditional relational databases and non-relational databases to various cloud storage and real-time stream data sources. To support digital operations, business analysis, and decision-making, enterprises need to efficiently integrate these heterogeneous data sources. This requires the use of a scheduling system to solve various problems brought by the intricate dependencies in the data processing technology (extraction transformation loading, ETL) of the data research and development data warehouse. The scheduling system has built-in basic data source access capabilities. However, the built-in basic data source access capabilities of the scheduling system only support a small number of general data sources (hereinafter referred to as native data sources). For details about native data sources, please refer to the following content and will not be elaborated here. Thus, in practical applications, there are often problems of being unable to directly access some special or new data sources, which affects the efficiency and timeliness of enterprise data processing.

[0035] In the embodiments of this application, the scheduling system is illustrated by taking a distributed task scheduling system as an example.

[0036] Taking enterprise resource planning (ERP) as an example, as Figure 1 shown, ERP is an enterprise information management system that mainly integrates material resources, financial resources, and information resources for the manufacturing industry. Its application and domestic substitution are one of the current information technology trends. A large amount of business data is generated during the daily operation of ERP, such as production order data, raw material inventory data, customer accounts receivable data, etc. These business data can be stored in the data sources of ERP, such as the self-developed databases supporting some domestic ERPs. The core role of the scheduling system is to orchestrate the enterprise data processing processes (such as data extraction, transformation, analysis, etc.). For example, in the scenario of synchronizing the business data generated by ERP to the data warehouse, the scheduling system needs to first access the data source of ERP, read the business data from it, and then synchronize the business data to the data warehouse, ultimately supporting subsequent report analysis and decision-making support. Or in the data migration scenario, the scheduling system needs to first access the data source of the old system (such as ERP1), read the data from it, and also access the data source of the new system (such as ERP2), and migrate the data read from the data source of the old system to the data source of the new system. It can be understood that the data source is the core object that the scheduling system needs to access, that is, the data source that the scheduling system needs to access.

[0037] However, since the scheduling system only supports a limited number of general data sources, it may not be compatible with ERP data sources. Specific incompatibility with ERP data sources, along with their unique protocols and dynamically changing requirements (such as switching from the old ERP data source to the new ERP data source during ERP localization), can lead to data processing interruptions. This necessitates a solution that can flexibly adapt to and efficiently access various data sources (especially ERP-specific data sources) to ensure a smooth transition of the data processing system and agile business switching.

[0038] This application provides a method for accessing a data source, applied to a first computing device. The method includes: First, receiving a first task, the first task indicating access to a first data source, the first task including a type item of the first data source, the type item including identification information of the first data source, the identification information of the first data source being used by the first computing device to obtain connection information of the first data source from a second computing device. Second, based on the identification information of the first data source, obtaining the connection information of the first data source from the second computing device. Then, based on the connection information of the first data source, obtaining a data source connection of the first data source, the data source connection of the first data source representing a communication channel between the first computing device and the first data source. Finally, based on the data source connection of the first data source, accessing the first data source.

[0039] Thus, based on the type item of the first data source included in the first task, the identification information of the first data source is obtained, and the connection information is obtained from the second computing device based on the identification information to create a data source connection for the first data source. There is no need to develop adaptation logic separately for each data source or manually configure the connection information. This simplifies the operation process of accessing the first data source, flexibly adapts to and efficiently accesses various data sources, and improves the flexibility and efficiency of data source access.

[0040] The system architecture of the embodiments of this application will be described below as an example.

[0041] like Figure 2 As shown in the illustration, this application provides a first computing device. The first computing device is capable of communicating with a second computing device.

[0042] A first computing device is configured to receive a first task, which instructs access to a first data source. The first task includes a type item for the first data source, which includes identification information of the first data source. This identification information is used by the first computing device to obtain connection information of the first data source from a second computing device. The first computing device is also configured to obtain the connection information of the first data source from the second computing device based on the identification information. Furthermore, the first computing device is configured to obtain a data source connection for the first data source based on the connection information, whereby the data source connection represents a communication channel between the first computing device and the first data source. Additionally, the first computing device is also configured to access the first data source based on the data source connection.

[0043] In some embodiments, the second computing device includes a correspondence between identification information of the first data source and connection information of the first data source. The second computing device receives the identification information of the first data source sent by the first computing device, determines the connection information of the first data source corresponding to the identification information based on the correspondence, and returns the connection information of the first data source to the first computing device. In this way, the first computing device can obtain the connection information of the first data source.

[0044] In one implementation, the second computing device includes a data source configuration module and a data source management module. The data source configuration module provides functions for configuring the identification and connection information of the first data source. The data source management module stores and manages the identification and connection information of the first data source.

[0045] In one implementation, the second computing device provides an interface for querying data, so that other devices (such as the first computing device) can obtain the connection information of the first data source from the second computing device through the interface. Specifically, the connection information of the first data source can be obtained from the data source management module of the second computing device.

[0046] In one implementation, the second computing device and the first computing device can be the same device or different devices (as described above). Figure 2 (As shown).

[0047] The aforementioned data source management module and data source configuration module can be deployed in the data hub of the computing device. The data hub is used to coordinate and manage the data source management module and the data source configuration module. For example, in an embodiment where the first computing device and the second computing device are different devices, the aforementioned data source management module and data source configuration module can be deployed in the data hub of the second computing device.

[0048] In one implementation, the first computing device obtains the interface configuration information of the second computing device based on a preset configuration file. The interface configuration information includes the interface address and interaction method of the second computing device. The configuration file includes the interface configuration information of the second computing device.

[0049] Specifically, in one implementation method where a computing device obtains connection information of a first data source from a second computing device based on the identification information of a first data source, the computing device obtains the connection information of the first data source from the second computing device based on the identification information of the first data source and the interface configuration information of the second computing device.

[0050] In this embodiment, the data source configuration module and data source management module deployed on the second computing device are mainly used to configure the identification information and connection information of the first data source. In this way, the first computing device can obtain the connection information of the first data source based on the second computing device, thereby achieving standardized configuration and efficient access to multiple types of heterogeneous data sources on the first computing device.

[0051] In this embodiment, the type of the first data source can be identified by FOUNDATION. Correspondingly, the first data source can also be called a FOUNDATION type data source. The first data source can include a native data source or a data source other than a native data source.

[0052] In the embodiments of this application, such as Figure 2 As shown, the first computing device is equipped with a scheduling system. Correspondingly, the second computing device and the first computing device are the same device, meaning that the aforementioned scheduling system and the data source configuration module and data source management module of the second computing device are deployed on the same computing device.

[0053] As mentioned earlier, native data sources are data sources supported by the basic data source access capabilities built into the scheduling system. For example, native data sources may include, but are not limited to, one or more of the following: relational database management systems, data warehouse tools, computing engines, columnar databases, data query engines, remote dictionary services, analytical databases, and in-memory databases.

[0054] For example, data sources other than native data sources can include, but are not limited to, one or more of the following: Elasticsearch search server, file transfer protocol (FTP), secure shell protocol (SSH), secure shell file transfer protocol (SFTP), and application programming interface (API).

[0055] In some embodiments, the scheduling system includes a data source plugin and an execution node service. The execution node service includes a task execution component. This can be understood as an inclusive relationship where the execution node service and the task execution component are responsible for carrying out and executing tasks. Specifically, the execution node service acts as the execution core, and the first computing device executes specific task logic by calling the task execution component. The task execution component is a functional plugin that the execution node service depends on when executing tasks; the two exhibit an inclusive collaborative relationship. The task execution component is used to execute a first task based on a first task execution instruction.

[0056] In one implementation where a computing device obtains connection information of a first data source from a second computing device based on the identification information of the first data source, the first computing device creates a channel between the first data source and the task execution component through a data source plugin. A client is created for the first data source through the data source plugin. The connection information of the first data source is then obtained from the second computing device based on the identification information of the first data source through the data source plugin.

[0057] In one implementation of obtaining the data source connection of the first data source based on the connection information of the first data source, the data source plugin obtains the data source connection of the first data source based on the connection information of the first data source, and returns the data source connection of the first data source to the task execution component through a channel based on the client.

[0058] In this embodiment, a data source plugin creates a channel and a client. Based on identification information, it obtains connection information from a second computing device and creates a data source connection for the first data source. Then, the client and channel return the data source connection to the first data source to the task execution component, enabling it to access the first data source. This expands the data sources accessible to the scheduling system without modifying its core logic, allowing seamless integration with heterogeneous data sources. Furthermore, access to the first data source is completed simply by obtaining connection information using identification information, without modifying configuration files. This preserves the original functionality of the scheduling system, shields underlying differences, achieves seamless data integration, improves the flexibility of data source access, enables rapid access to multiple types of data sources, and increases the success rate of task execution by the scheduling system. In the embodiments of this application, such as Figure 3 As shown, the task execution component may include one or more of the following executors, with different executors used to support different task types: An open-source distributed big data processing engine executor used to execute distributed data stream processing or batch processing jobs. If the job involves data reading and writing, it may access a relational database management system to achieve data processing.

[0059] A shell executor is used to execute shell scripts. If the script involves data operations, it may access a relational database management system to complete data reading and writing.

[0060] A Structured Query Language (SQL) executor is used to execute SQL statements, such as queries, CRUD operations, etc. It typically accesses relational database management systems, data warehouse tools, computing engines, etc.

[0061] The Sub_Flow executor is used to nest and call other defined workflows, enabling workflow reuse.

[0062] The stored procedure executor is used to call built-in stored procedures in relational database management systems, columnar databases, analytical databases, and other similar systems.

[0063] Computational models, frameworks, and platforms (MapReduce, MR) executors for parallel big data processing, which are used to submit MR parallel computing jobs and may access data warehouse tools, etc., to process big data tasks.

[0064] The Python executor is used to execute Python scripts; if the script involves data reading and writing, it may access relational database management systems, etc.

[0065] The Dependent executor is used to configure dependencies between tasks (such as waiting for other tasks to complete before execution).

[0066] The Hypertext Transfer Protocol (HTTP) executor is used to send HTTP requests, such as calling external APIs.

[0067] In this embodiment of the application, the task execution component is described using an SQL executor as an example.

[0068] In one implementation, the scheduling system includes a native data source configuration module and a data source management module. To distinguish them from the data source configuration module and data source management module of the first computing device, they are referred to as the native data source configuration module and the native data source management module, respectively. The native data source configuration module is used to configure the connection information of the native data sources supported by the scheduling system, while the native data source management module is used to store and manage the connection information of the native data sources.

[0069] In some embodiments, the first computing device is further configured to receive a second task, the second task indicating access to a second data source, the second task including a type item for the second data source, the type item including connection information of the second data source. The first computing device is further configured to determine the connection information of the second data source based on the second task, and to obtain the data source connection of the second data source based on the connection information. The data source connection of the second data source represents a communication channel between the first computing device and the second data source. Furthermore, the first computing device is also configured to access the second data source based on the data source connection of the second data source.

[0070] For example, the second data source can be the original data source mentioned above.

[0071] In one implementation, the first computing device creates a channel between the second data source and the task execution component via a data source plugin. A client is created for the second data source via the data source plugin. The data source plugin obtains the data source connection of the second data source based on its connection information and returns this connection to the task execution component through the channel based on the client.

[0072] It should be understood that this scheduling system supports flexible selection of solutions based on different types of data sources (such as a second data source or a first data source): If the task indicates that the data source to be accessed is a second data source natively supported by the scheduling system (such as a relational database management system, data warehouse tools, etc.), the solution of directly configuring connection information can be adopted (i.e., the processing flow of the second task), and the corresponding data source connection can be obtained and accessed through the data source plugin of the scheduling system. Alternatively, regardless of whether the data source to be accessed in the task is a second data source natively supported by the scheduling system or a data source other than the native data source (such as Elasticsearch), the solution of obtaining connection information through a second computing device can be adopted (i.e., the processing flow of the first task), and the connection information can be obtained from the second computing device with the help of the data source plugin, achieving seamless access to heterogeneous data sources. In this way, the scheduling system can retain efficient support for native data sources, and can also be compatible with various non-native data sources through extended solutions, meeting the access and management needs of multiple types of heterogeneous data sources.

[0073] In some embodiments, such as Figure 3As shown, the scheduling system also includes a scheduler. The scheduler receives the target task (such as a first task or a second task) and generates a corresponding task execution instruction based on the target task (such as a first task execution instruction for the first task, or a second task execution instruction for the second task). This task execution instruction is used to instruct the execution of the target task. The scheduler also sends the task execution instruction to a data queue. For example, the data queue can be a distributed queue implemented based on a remote dictionary service. When the scheduler sends the task execution instruction to the data queue, the execution of the target task is triggered. For example, there may be multiple schedulers, and their collection can be called the scheduling core layer.

[0074] In this embodiment of the application, the target task can be an SQL type task.

[0075] In some embodiments, such as Figure 3 As shown, the scheduling system includes multiple execution node services. Execution node services that are not currently executing tasks monitor the data queue. When the scheduler sends a task execution instruction to the data queue, any one of the execution node services that is not currently executing a task retrieves the instruction from the data queue to execute the target task. For example, the collection of multiple execution node services can be referred to as the execution core layer.

[0076] In some embodiments, after the execution node service obtains the first task execution instruction, it creates a task execution context for the first task. The task execution context of the first task is used to store information required by the task execution component to execute the first task, including the identification information of the first data source and the interface configuration information of the second computing device.

[0077] In one implementation, after the execution node service receives the first task execution instruction, the first computing device creates a target thread for the first task. This target thread is used to execute the task execution component; essentially, it serves as the runtime environment for the task execution component, which executes the first task within it. Thus, after the target thread starts, the task execution component retrieves the identification information of the first data source and the interface configuration information of the second computing device from the task execution context of the first task. It then sends these information to the data source plugin. Subsequently, the various components of the data source plugin begin operating, enabling the task execution component to obtain a data source connection to the first data source, thereby achieving access to the first data source.

[0078] In some embodiments, the scheduling system interacts with the data development module. The data source development module orchestrates target tasks and sends them to the scheduling system. Specifically, the data development module provides a user interface for users to input target tasks through the interface provided by the data development module to complete the visual configuration of the target tasks.

[0079] For example, the data source development module can be deployed on the first computing device or on other devices (such as the first computing device).

[0080] like Figure 3 As shown, the scheduling system also includes a call interface, a distributed coordination layer, a data persistence layer, and an alarm layer. The call interface interacts with the data development module, retrieves the target task from the data development module, and then transmits it to the scheduler. The distributed coordination layer is the coordination center of the scheduling system, supporting service registration and heartbeats. For example, the scheduler and execution node services register with it upon startup and periodically send heartbeats to maintain their liveness. The distributed coordination layer achieves fault tolerance by monitoring node status and triggers rescheduling when a node fails. The data persistence layer stores the scheduling system's metadata, including workflow definitions, task definitions, task instances, execution status, and historical logs. The alarm layer retrieves abnormal task execution states (such as task failure or timeout) from the data persistence layer and sends alarm notifications according to configured alarm rules (email, SMS, etc.).

[0081] In this embodiment, the first computing device and / or the second computing device can be a server. The server can be a single physical server or logical server, or it can be composed of two or more physical servers or logical servers that share different responsibilities, working together to achieve various server functions such as data processing and service provision.

[0082] In terms of hardware form, servers can be blade servers, high-density servers, rack servers, or tower servers, which are suitable for different application scenarios such as high-density cluster deployment in data centers and small enterprise server rooms.

[0083] like Figure 4 As shown, the hardware of a computing device includes a processor, a basic input / output system (BIOS) chip, a baseboard management controller (BMC), and memory, while the software mainly includes BIOS firmware, an out-of-band management module, and an operating system (OS).

[0084] A processor may include a central processing unit (CPU). A CPU includes one or more CPU cores, each containing an arithmetic unit, a control unit, and a register set. It supports instruction sets such as integer and floating-point arithmetic. All data processing operations are executed by the CPU cores. Under the same processor architecture (e.g., x86, ARM, RISC-V) and manufacturing process, a higher number of CPU cores generally results in stronger parallel data processing capabilities, theoretically accelerating data processing speed. In the embodiments of this application, the CPU can run a scheduling system.

[0085] The BIOS chip, located in a chip slot on the motherboard, is an important piece of hardware during the boot process of a computing device. It may be an electrically erasable programmable read-only memory (EEPROM) or flash memory chip, used to store BIOS firmware, BIOS initialization parameters, and BIOS detection results of components in the computing device.

[0086] The program code for the out-of-band management module can be stored in dedicated flash memory (such as the flash memory area built into the BMC chip, the flash memory area in the BIOS chip, etc.). This type of flash memory is dedicated to storing the program code for the out-of-band management module. When it is stored in the BIOS chip, it is separate from the flash memory area in the BIOS chip used to store BIOS firmware and related parameters, and they do not occupy space together. It should be noted that the specific form of the out-of-band management module in this application embodiment is not limited; the above is only an illustrative example.

[0087] Memory, also known as internal memory or main memory, is installed in memory slots on the motherboard of a computing device and connected to the CPU via the memory bus. It is primarily used for temporary storage of data and instructions, providing high-speed data read and write support for the processor. For example, during device operation, frequently interacting content such as operating system and application runtime data, and temporary calculation results are temporarily stored in memory.

[0088] In addition, computing devices are typically equipped with storage devices such as hard drives for persistent storage of operating system images, application code, business data, etc. Interfaces integrated on the motherboard, such as Universal Serial Bus (USB) and Peripheral Component Interconnect Express (PCIe), can be used to connect peripherals (such as display interface cards, network interface cards, RAID cards, and encryption cards) to enhance device functionality.

[0089] BIOS firmware is a set of programs embedded in the BIOS chip. It runs first when the computing device is powered on, completes hardware initialization and detection, and then boots the OS, allowing users to use the computing device normally. It provides the lowest-level and most direct hardware settings and control for the computing device, such as setting the hard drive boot order and adjusting CPU power consumption.

[0090] An operating system (OS) is a computer program that manages and controls the hardware and software resources of a computing device. It runs on the processor, relies on hardware such as memory to work together, and is responsible for tasks such as process management, memory allocation, file system management, and device driver loading. It is the foundation for users to operate devices and for other software to run.

[0091] The OS program files are stored on external storage devices. Here, "external" refers to storage devices independent of the motherboard's core circuitry, as opposed to the CPU, memory, and other components of the computing device. Common types include hard disk drives (HDDs) and solid-state drives (SSDs) that are directly connected to the motherboard's storage interface. These devices are used for persistent storage of large amounts of data (such as operating system images, applications, and business data). During runtime, the operating system moves the program files from the external storage device to memory, where the processor reads and executes the instructions.

[0092] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0093] For ease of understanding, the data source access method provided in the embodiments of this application will be described exemplarily below with reference to the first computing device and the accompanying drawings.

[0094] It is understandable that the first data source can be configured before accessing it, and a component for accessing the first data source can be deployed in the scheduling system. Therefore, this application embodiment will describe the data source access method in three parts.

[0095] Part One: Combination Figure 5 , Figure 6 and Figure 13 This section introduces the specific implementation method for configuring the first data source.

[0096] Part Two: Combination Figure 7 This section introduces the specific implementation method for deploying data source plugins.

[0097] Part Three: Combination Figure 8This section describes the specific implementation method of configuring the type item of the first data source in the scheduling system.

[0098] Part Four: Combination Figure 9 , Figure 10 , Figure 11 , Figure 12 and Figure 13 This section introduces the specific implementation method for accessing the first data source.

[0099] Part One, such as Figure 5 As shown in the embodiments of this application, a specific implementation method for configuring a first data source is provided. Configuring a data source can refer to: determining the location of the first data source, setting the basic attributes and rules of the first data source (such as labeling the type of the first data source to match the corresponding protocol). For example, it includes the following steps: S501-S503, which can be executed by the processor of the second computing device, such as a CPU.

[0100] S501 receives the connection information and identification information of the first data source input by the user.

[0101] In this embodiment of the application, the type item of the first data source can be FOUNDATION.

[0102] In some embodiments, the first data source includes data sources other than the aforementioned native data source, and may also include the native data source. That is, the data source access method provided in this application embodiment can access data sources other than the native data source, or it can access the native data source, and this is not limited here. It should be noted that in this embodiment, the native data source is also referred to as FOUNDATION.

[0103] For example, the connection information includes the address of the target server where the data source is located, the port number of the port on the target server where the data source provides services to the outside world, and the account and password for accessing the data source.

[0104] For example, the identification information of the first data source includes the name and / or identifier (such as an identity document, ID). The identifier of the first data source can be represented by a catalog, and the name of the first data source can be represented by a schema. In this embodiment, the identification information may include both an identifier and a name. This is because multiple data sources within the first data source may share the same IP address. For example, the target server's IP address might be 192.168.1.***, on which two independent relational database management systems, "e-commerce order database" and "member user database," are deployed. In this case, relying solely on the IP address would only locate the relational database management system on that server, and would not determine whether to connect to the e-commerce order database or the member user database. Thus, using both identifier and name—two multi-dimensional pieces of information—better meets the need for obtaining connection information and achieves effective location.

[0105] In one implementation, the second computing device configures the connection information and identification information of the first data source through a data source configuration module.

[0106] For example, the data source configuration module stores an information template that includes configuration items used to configure the identification information of different data sources in the data source of the FOUNDATION type item. The information template also includes connection information items, which include connection information for the different data sources in the data source of the FOUNDATION type item.

[0107] For example, users can fill in the corresponding information according to the configuration items and connection information items through the visual interactive interface (such as a World Wide Web form) provided by the data source configuration module. During the filling process, the data source configuration module can verify the legality of the entered information in real time (such as whether the port number is a valid number and whether the account password meets the complexity requirements). Only after the verification is passed can the information be submitted. After submission, the data source configuration module can obtain the connection information and identification information entered by the user through the aforementioned visual interactive interface. The data source configuration module can also send the obtained connection information and identification information to the data source management module of the second computing device for storage and management of the connection information and identification information. The connection information and identification information of the data source have a corresponding relationship.

[0108] S502 stores the connection information and identification information of the first data source.

[0109] In one implementation, the second computing device stores the connection information and identification information of the first data source received through the data source configuration module in the data source management module.

[0110] S503, associate the identification information, connection information and type item of the first data source.

[0111] In one implementation, the second computing device associates the identification information, connection information, and type item of the first data source with the data source management module.

[0112] In this embodiment of the application, by associating the identification information and connection information of the first data source with the type of the first data source, it is easier to obtain the connection information of the first data source in the future, thereby creating a data source connection for the first data source.

[0113] In one implementation, the second computing device can also record the submission time of the connection information of the first data source and the configuration status of the configuration items (such as whether the configuration status can be enabled or disabled) through the data source management module.

[0114] In one implementation, the second computing device can also verify the existence of a first data source with the same type, the same identifier, and the same name through the data source management module, thus avoiding duplicate configuration.

[0115] In one implementation, the data source management module can also display the first data source. Specifically, the data source management module displays the data source (i.e., the newly added data source) of the FOUNDATION type item through a visual interactive interface. For example, as shown... Figure 6 As shown, new data sources can include five main categories: relational databases, analytical databases, distributed system infrastructures, messaging systems, and other types of databases. For example, other types of databases can include Elasticsearch, FTP, SFTP, and API.

[0116] In some embodiments, the second computing device is capable of interacting with other devices through its interface.

[0117] The interface of the second computing device is used to query data. For example, the interface of the second computing device includes interface 1, which is specifically the interface of the data source management module. Interface 1 is used by other devices (such as the first computing device) to query the connection information of the first data source. In one implementation, interface 1 can be adapted to the execution node service of the scheduling system; that is, the execution node service can call interface 1 to obtain the connection information of the first data source.

[0118] In this embodiment of the application, the above-described process of configuring the first data source ensures the integrity of the connection information of the data source of the FOUNDATION type item by using the templated configuration classified by data source type, thus avoiding subsequent connection failures due to missing or incorrect parameters. On the other hand, by associating the connection information, identification information and data source type, data security is ensured and the first data source can be accurately located.

[0119] Example, combination Figure 13 As shown, firstly, the first data source is configured through the data source configuration module. Secondly, the connection information and identification information of the first data source are stored in the data source management module. Furthermore, the execution node service can query the connection information of the first data source through interface 1.

[0120] Part Two, such as Figure 7 As shown, this application embodiment provides a specific implementation method for deploying a data source plugin. For example, it includes the following steps: S701-S702, which can be executed by the processor of the first computing device, such as the CPU.

[0121] S701, retrieve the first component, the second component, the third component, and the fourth component.

[0122] In one implementation, the third component is used to obtain the identification information of the first data source. The first component is used to create a channel between the first data source and the task execution component, and to create the second component as a client of the first data source. The fourth component is used to obtain the identification information of the first data source from the third component, and based on the identification information, to obtain the connection information of the first data source from the second computing device, and based on the connection information, to obtain the data source connection of the first data source, and to return the data source connection of the first data source to the client of the first data source. The client of the first data source is used to return the data source connection of the first data source to the task execution component through the aforementioned channel.

[0123] In one implementation, the third component can also be used to obtain interface configuration information of the second computing device. The interface configuration information includes the interface address and interaction method of the second computing device. For example, the interface configuration information can be the Uniform Resource Locator (URL) of the interface. That is, the interface configuration information indicates which interface of the second computing device (e.g., interface 1) is called, and the interaction method with that interface.

[0124] Accordingly, in the above embodiments, the fourth component is specifically used to obtain the identification information of the first data source and the interface configuration information of the second computing device (specifically, the interface configuration information of interface 1) from the third component, and based on the identification information of the first data source and the interface configuration information of the second computing device, obtain the connection information of the first data source from the second computing device, create a data source connection of the first data source based on the connection information of the first data source, and return the data source connection of the first data source to the client of the first data source.

[0125] In one implementation, the first, second, third, and fourth components mentioned above are predefined classes, each including fixed logic and blank attributes. The fixed logic and blank attributes differ between components. For example, the fixed logic can represent that each predefined class has its "how to do it" logic (e.g., how the channel is created, how the client is initialized) predefined during deployment. Blank attributes can represent that each predefined class reserves space for "what parameters to use" (e.g., member variables such as the identifier information of the first data source in the class). For instance, when creating a client for the second component as the first data source, the blank attributes in the second component are set to the parameters corresponding to the first data source (e.g., the identifier information of the first data source), thereby enabling the creation of the second component as a client for the first data source.

[0126] In one implementation, the user writes code for a first component, a second component, a third component, and a fourth component, and stores the code in a first computing device so that the first computing device can access the first component, the second component, the third component, and the fourth component.

[0127] S702 loads the first, second, third, and fourth components into the data source plugin.

[0128] In one implementation, the first, second, third, and fourth components are loaded into the data source plugin during the deployment and startup of the scheduling system. In this implementation, each loaded component includes fixed logic and blank attributes, already possessing the logic for "how to do it" (e.g., how to create the channel, how to initialize the client). For example, the first component already has the logic for creating the channel between the first data source and the task execution component, but the specific parameters of the first data source associated with the channel (e.g., the type of the first data source) and other attributes (i.e., blank attributes) still have default values ​​(e.g., null). This avoids recreating the plugin class and compiling the fixed logic every time a data source access task is executed, reducing resource overhead during task execution and improving response speed.

[0129] In this embodiment, the data source plugin is extended by loading the first, second, third, and fourth components. Furthermore, by pre-deploying these components during the scheduling system's deployment and startup, the operations of recreating the plugin class and compiling fixed logic each time a data source access task is executed are avoided, effectively reducing resource overhead during task execution and improving response speed. Simultaneously, the clear division of labor among the components ensures the orderly connection between the first data source and the task execution component, guaranteeing the efficiency and reliability of data source access without requiring modification to the core logic of the scheduling system.

[0130] In some embodiments, the data source plugin further includes a fifth component, a sixth component, a seventh component, and an eighth component. The seventh component is used to obtain connection information of the second data source. The fifth component is used to create a channel between the second data source and the task execution component, and to create the sixth component as a client of the second data source. The eighth component is used to obtain the connection information of the second data source from the seventh component, obtain the data source connection of the second data source based on the connection information, and return the data source connection of the second data source to the client of the second data source. The client of the second data source is used to return the data source connection of the second data source to the task execution component through the aforementioned channel.

[0131] It should be understood that the fifth, sixth, seventh, and eighth components mentioned above are native components in the data source plugin, used to enable access to the second data source.

[0132] Part Three, such as Figure 8 As shown, this application embodiment provides a specific implementation method for configuring the type item of the first data source in the scheduling system. For example, it includes the following steps: S801-S802, which can be executed by the processor of the first computing device, such as the CPU.

[0133] S801 responds to the user's selection of the first data source type by displaying the configuration interface.

[0134] The configuration interface includes a first option, which is used by the user to configure the identification information of the data source.

[0135] The following is an example of how the configuration interface is displayed in response to the user's selection of the first data source type: On the "Create New Data Source" page of the scheduling system, users click the "Data Source Type" drop-down menu and select the "First Data Source Type" (e.g., "FOUNDATION"). The scheduling system then monitors this selection and displays the configuration interface. This interface shows the first option, used by the user to configure the data source identifier or name, and also displays the first data source type.

[0136] Different types of data sources require different configuration items. For example, the configuration item for the first data source is its identification information, while the configuration item for the second data source is its connection information. Therefore, the configuration interface here includes a first option to allow the user to configure the identification information of the first data source.

[0137] Before displaying the configuration interface in response to a user's selection of the first data source type, the operator (someone with development privileges on the scheduling system) modifies the scheduling system's code to add the first data source option to the system's native data source management module. A dedicated front-end form is designed for the first data source, including its identification information. Thus, when the scheduling system detects a user's selection of the first data source type, it dynamically renders the front-end form into a visual configuration interface, allowing the user to fill in the first data source's identification information.

[0138] S802 receives the identification information of the first data source configured by the user in the first option.

[0139] In this embodiment, by extending the native data source management module of the development side of the scheduling system to configure the identification information of the first data source, it is possible to not only add the first data source type item, but also dynamically render the matching configuration interface when the user selects the type, and accurately receive the identification information configured by the user. This provides key input for obtaining connection information from the second device based on the identification information of the first data source, thereby realizing the seamless extension of the scheduling system to heterogeneous data sources and improving the flexibility and ease of use of the scheduling system.

[0140] Part Four, such as Figure 9 As shown, this application embodiment provides a specific implementation method for accessing the first data source. For example, it includes the following steps: S901-S905, which can be executed by the processor of the first computing device, such as the CPU.

[0141] S901 retrieves the interface configuration information of the second computing device from the preset configuration file.

[0142] The configuration file includes the interface configuration information for the second computing device. The configuration file is stored in the scheduling system.

[0143] In one implementation, when the scheduling system starts, the first computing device reads the interface configuration information of the second computing device from a preset configuration file through the execution node service. Understandably, the execution node service also starts when the scheduling system starts. After obtaining the interface configuration information of the second computing device, the execution node service can also write the interface configuration information of the second computing device into its local memory.

[0144] It should be noted that the execution node service here can be any execution node service in the scheduling system, such as execution node service 1.

[0145] In one implementation, if the configuration file is updated, the execution node service reads the latest interface configuration information of the second computing device from the configuration file to ensure that the interface configuration information of the second computing device takes effect in real time. For example, if the configuration file is re-stored in the scheduling system after the execution node service obtains the interface configuration information of the second computing device, then the configuration file can be considered updated.

[0146] Step S901 above is an optional step.

[0147] S902, receive the first task.

[0148] The first task indicates access to the first data source. The first data source can include one or more data sources. For example, the first task could be an SQL-type task.

[0149] For example, the first task includes specific operational requirements for accessing the first data source (such as executing an SQL query), and a type field for the first data source, which includes identification information of the first data source. The identification information of the first data source is used by the first computing device to obtain connection information from the second computing device. If the first task is an SQL-type task, it may also include task execution parameters, such as SQL statements.

[0150] In one implementation, the first task is received through a data development module. The user inputs the first task on the interactive interface provided by the data development module, and the data development module stores the first task in the scheduling system through a call interface. Thus, the scheduling system receives the first task. (Continuing with the example...) Figure 13 To clarify, the data development module is deployed on the first computing device. The first computing device receives the first task input by the user through the data development module and starts the scheduling system, specifically by initiating the scheduling system's API call. Furthermore, the data development module sends the first task to the scheduling system so that the scheduling system can receive the first task.

[0151] In another implementation, the first task is received by the user through a scheduling system. The scheduling system provides a visual interface where the user can directly input the first task to receive it.

[0152] Example, combination Figure 10As shown, users can input the first task in the content input area of ​​the interactive interface provided by the data development module. The content of the first task includes: the type of the first task (e.g., SQL task), the identifier of the creator of the first task (e.g., IW00*****), the creation time of the first task (e.g., 2025 / 03 / 03 18:17:20 GMT+08:00), the custom parameter setting format (e.g., ${}, example: ${task execution parameter name}, and supports system-configured parameters), the specific operational requirements for accessing the first data source, and the type item (FOUNDATION) of the first data source.

[0153] S903, based on the identification information of the first data source, obtains the connection information of the first data source from the second computing device.

[0154] In one implementation method for obtaining connection information of a first data source from a second computing device based on the identification information of the first data source, the connection information of the first data source is obtained from the second computing device based on the identification information of the first data source and the interface configuration information of the second computing device. Thus, by first obtaining the interface configuration information of the second computing device from a preset configuration file, and then combining it with the identification information of the first data source to obtain the connection information from the second computing device, unified management and flexible adjustment of the interface configuration are achieved, while ensuring the standardization and accuracy of the obtained connection information of the first data source, thereby improving the reliability and maintainability of the data acquisition process.

[0155] Specifically, based on the interface configuration information of the second computing device, the identification information of the first data source is sent to the second computing device. The connection information of the first data source returned by the second computing device is received. Correspondingly, after receiving the identification information of the first data source, the second computing device determines the connection information of the first data source based on the correspondence between the identification information and the connection information, and returns it to the first computing device.

[0156] In one possible implementation, the first computing device can specifically obtain the connection information of the first data source from the second computing device based on the identification information of the first data source by coordinating the work of various components in the scheduling system. For example, this includes the following steps: S1101-S1109.

[0157] S1101, the scheduler generates the first task execution instruction based on the first task.

[0158] For example, the first task execution instruction includes: the type of the first data source, task execution parameters (such as SQL statements), etc. In one implementation, the first computing device obtains the first task through the scheduler, parses the first task, and generates the first task execution instruction corresponding to the first task.

[0159] For example, the execution instructions for the first task may also include: execution resource limitations (such as the number of CPU cores, memory size, etc.). The execution resource limitations can be configured by the user when orchestrating the first task. For instance, the user inputs the scheduling information corresponding to the first task through the interactive interface provided by the data development module. The first computing device obtains this scheduling information and generates the execution instructions for the first task based on the first task and the scheduling information. The scheduling information may include execution resource limitations, etc. For example, continuing to combine... Figure 13 The explanation is as follows: scheduling information is obtained through the data development module. Accordingly, in one implementation, the first computing device obtains the first task and scheduling information through the scheduler, parses the first task and scheduling information, and generates the first task execution instruction corresponding to the first task.

[0160] S1102, the scheduler sends the first task execution instruction to the data queue.

[0161] In one implementation, the first computing device can also mark the initial state of the first task execution instruction through a scheduler, such as the initial state including "to be executed," to indicate that the first task execution instruction is in a pending execution state. At this time, if the scheduler sends the first task execution instruction to the data queue, the execution of the first task is triggered.

[0162] For example, continue to combine Figure 13 To explain, the scheduler obtains the first task and scheduling information based on the scheduling system's call interface, generates the first task execution instruction corresponding to the first task, and sends the first task execution instruction to the data queue.

[0163] S1103, retrieves the first task execution instruction from the data queue by executing the node service.

[0164] In one implementation, the execution node service is an execution node service that has not yet executed any tasks. It can be execution node service 1 as described above, or it can be an execution node service other than execution node service 1, such as execution node service 2. For example, the execution node service monitors the data queue at preset intervals to see if there are any marked first task execution instructions waiting to be executed. If so, it retrieves the first task execution instruction from the data queue. This achieves balanced distribution and timely scheduling of the first task execution instructions.

[0165] S1104, in response to the first task execution instruction, creates a corresponding first task execution context for the first task through the execution node service, and stores the key parameters of the first data source into the first task execution context.

[0166] The key parameters include the identification information of the first data source. For example, the key parameters may also include the interface configuration information of the second computing device.

[0167] In one implementation, the task execution context is initialized by executing the node service, thereby creating a first task execution context for the first task.

[0168] For example, continue to combine Figure 13 To explain, after the scheduler sends the first task execution instruction to the data queue, the execution node service obtains the first task execution instruction and executes the first task based on the first task execution instruction.

[0169] S1105, obtain the key parameters of the first data source from the first task execution context through the task execution component.

[0170] For example, if the first task is an SQL task, the corresponding task execution component could be an SQL executor.

[0171] S1106 passes key parameters of the first data source to the data source plugin through the task execution component.

[0172] Specifically, the task execution component passes the key parameters of the first data source to the third component of the data source plugin. For example, by setting the blank properties in the third component to the identification information of the first data source and the interface configuration information of the second computing device, the key parameters of the first data source can be passed to the data source plugin, that is, the key parameters of the first data source can be obtained through the third component.

[0173] In the above embodiment of this application, by creating a first task execution context for the first task and storing the key parameters of the first data source, and then having the task execution component obtain and pass them to the data source plugin (such as the third component), the unified storage and orderly transmission of the key parameters of the first data source is achieved. This not only avoids confusion and loss of key parameters when they are transmitted between components, but also ensures that the data source plugin can accurately obtain the key parameters, providing stable data transmission support for the collaborative completion of data source access by the various components of the data source plugin.

[0174] In other embodiments, in step S1106 above, the first computing device can transmit only the interface configuration information of the second computing device from the key parameters of the first data source to the data source plugin through the task execution component, while the identification information in the key parameters of the first data source can be determined by the configuration parameters input by the user. For example, the first computing device can receive the configuration parameters input by the user, which include the identification information of the first data source. The identification information of the first data source is then configured in the third component according to the configuration parameters. Thus, the identification information of the first data source is obtained through the third component.

[0175] For example, the scheduling system's visual interface also serves as a place for user-inputted configuration parameters. Figure 12As shown, the visual interface has elements for creating a data source, including a data source selection control (such as selecting FOUNDATION, which indicates a data source of type FOUNDATION), as well as input controls corresponding to the data source name and data source identifier, and provides an "OK" button. After the user clicks the "OK" button, the task execution component is triggered to pass the key parameters of the first data source to the data source plugin.

[0176] In this embodiment of the application, by receiving configuration parameters input by the user and configuring the identification information of the first data source in the third component, the source of the identification information obtained by the data source plugin can be directly identified, thereby improving the accuracy and flexibility of the identification information configuration.

[0177] In one implementation, configuration parameters input by the user are received through a visual interactive interface provided by the data source plugin, and the identification information of the first data source in the configuration parameters is stored in a third component. In this way, the data source plugin (specifically, the third component) obtains the identification information of the first data source.

[0178] S1107 creates a channel between the first data source and the task execution component through the data source plugin.

[0179] The channel between the first data source and the task execution component is used by the data source plugin to transmit the data source connection of the first data source to the task execution component.

[0180] In one implementation, a channel between the first data source and the task execution component is created by a data source plugin based on the type of the first data source.

[0181] In one implementation method that creates a channel between a first data source and a task execution component through a data source plugin, the first component is started, and the channel between the first data source and the task execution component is created through the first component. In this way, creating a channel between the first data source and the task execution component through the first component can adapt to the establishment of channels for data sources of different types, improving the standardization and scalability of data source access.

[0182] S1108 is a client that creates a first data source for a first data source through a data source plugin.

[0183] In one implementation of creating a client for a first data source through a data source plugin, the identification information of the first data source is obtained from the third component through the first component, and the second component is created as the client of the first data source based on the identification information of the first data source.

[0184] In this embodiment, the second component is created as a client for the first data source through the first component, laying the foundation for subsequent acquisition of connection information based on identification information and creation of data source connection, and further improving the standardization and scalability of data source access.

[0185] For example, by setting the blank property of the second component to the identifier information of the first data source through the first component, the second component is created as a client of the first data source.

[0186] S1109: Based on the identification information of the first data source, the data source plugin obtains the connection information of the first data source from the second computing device.

[0187] In one implementation that obtains connection information of a first data source from a second computing device based on the identification information of the first data source via a data source plugin, a fourth component obtains the connection information of the first data source from the second computing device based on the identification information of the first data source obtained from a third component and the interface configuration information of the second computing device.

[0188] In this embodiment of the application, the connection information of the first data source is obtained from the second computing device through multiple component protocols in the scheduling system, so as to facilitate the subsequent acquisition of the data source connection of the first data source.

[0189] S904: Based on the connection information of the first data source, obtain the data source connection of the first data source.

[0190] The data source connection of the first data source represents the communication channel between the first computing device and the first data source. The data source connection enables the task execution component to send data operation requests (such as SQL execution, data writing, etc.) to the data source and receive the operation results returned by the data source (such as query result sets, execution success / failure indicators, error codes, etc.). For example, the data source connection can be an instance, specifically an object instance.

[0191] In one implementation of obtaining the data source connection of a first data source based on its connection information, a first data source instance is obtained based on the first data source connection information. This first data source instance is used to obtain the data source connection of the first data source. The first data source instance is then invoked to obtain the data source connection of the first data source.

[0192] In this embodiment of the application, by first obtaining a first data source instance, since the data source instance has the function of obtaining a data source connection, the data source connection of the first data source can be obtained.

[0193] In one implementation, obtaining a first data source instance based on the connection information of a first data source includes: obtaining a target object and configuring the connection information of the first data source into the target object, whereby the target object encapsulates the connection information of the first data source; and calling a first method, using the target object configured with the connection information of the first data source as an input parameter to create a first data source instance, whereby the first method is used to create the first data source instance.

[0194] The target object is a wrapper for the connection information of the primary data source, such as the parameter class DataSourceParam in the utility library. By configuring the connection information of the primary data source into the target object one by one, it is ensured that the target object fully carries the parameters required for the connection of the primary data source, and the configuration information is transmitted uniformly to avoid parameter loss due to dispersion.

[0195] The first method represents the method for creating the first data source instance, such as DataSourceUtil.create(). When the first method is called, the target object with the configured connection information is passed as an input parameter; the first method internally parses the connection parameters in the target object and creates the first data source instance (such as DataSource), which has the ability to create and manage (such as reuse and recycle) data source connections.

[0196] In one possible implementation, invoking a first data source instance to obtain a data source connection for the first data source includes: invoking the first data source instance to create a data source connection for the first data source.

[0197] For example, the connection acquisition method provided by the first data source instance is called to cause the first data source instance to create a data source connection for the first data source. This connection acquisition method is used to create or obtain a data source connection for the first data source, such as GetConnection().

[0198] In this embodiment, it can be applied to scenarios where a first data source instance is created for the first time based on the connection information of the first data source. It can complete the entire process of configuring the target object, creating the first data source instance, and generating the data source connection during the initial access to the first data source, improving the standardization, reliability, and maintainability of the first data source access.

[0199] In another possible implementation, obtaining a first data source instance based on the connection information of the first data source includes: determining, from a set of data source instances, the data source instance corresponding to the connection information of the first data source as the first data source instance. The set of data source instances includes data source instances corresponding to the connection information of different data sources. The first data source instance includes multiple data source connections. The connection parameters of the multiple data source connections are different. The connection parameters include one or more of the following: number of connections, connection status, and lifecycle.

[0200] It should be understood that multiple data source connections will form a connection pool.

[0201] For example, the connection count includes the upper limit of the number of data source connections in the connection pool, the minimum number of idle data source connections, etc. An idle data source connection (also called an idle connection) indicates that the data source connection is not currently in use, such as when a task is being executed (e.g., the first task). Conversely, if the data source connection is currently in use, it can be marked as an active connection.

[0202] For example, connection status includes the data source connection acquisition timeout (in milliseconds) and the data source connection idle timeout. The data source connection acquisition timeout represents the maximum waiting time (e.g., a scheduling system) when requesting a data source connection from the connection pool; if the connection is not acquired within the timeout period, an error is thrown. The data source connection idle timeout represents the maximum unused lifespan of an idle data source connection in the connection pool; if the connection remains unused after the timeout period, it is closed or reclaimed.

[0203] For example, the lifecycle includes the maximum lifecycle of the data source connection (in milliseconds), which represents the longest time a data source connection can live from being created until it is forcibly closed or recycled.

[0204] In one possible implementation, invoking a first data source instance to obtain the data source connection of the first data source includes: invoking the first data source instance to determine the data source connection of the first data source from multiple data source connections based on connection parameters.

[0205] Specifically, the first data source instance is invoked. This instance first scans for idle data source connections, prioritizing those idle connections whose idle duration has not exceeded the idle timeout period and whose remaining lifecycle time is sufficient, and uses these as the primary data source connection. Alternatively, if the number of idle data source connections in the connection pool is insufficient and has not reached the maximum number of data source connections, a new data source connection is created and used as the primary data source connection.

[0206] For example, if the maximum number of data source connections is 20, it means that the connection pool can hold a maximum of 20 data source connections (including active and idle connections). Assuming that there are 8 requests executing SQL at this time, then these 8 data source connections used are "active connections", and the remaining 12 (if they have been created) are "idle connections". If a 9th request comes in, it will first use an idle connection. If there are no idle connections and the maximum number of connections has not been reached, a new connection will be created and marked as an "active connection".

[0207] In this embodiment, by accurately matching the first data source instance from the data source instance set and obtaining the data source connection of the first data source based on the connection parameters, the efficient reuse of the data source connection and resource consumption control are achieved, while ensuring the availability of the data source connection and improving the efficiency, stability and resource utilization of obtaining the first data source connection.

[0208] In one implementation, the main method can be used to retrieve the first data source instance and its connection. The main method, such as the `main()` method, provides the logical entry point for retrieving the first data source instance and its connection. The main method is housed within the main class, which serves as the code container for it.

[0209] In some embodiments, the method further includes: invoking a second method to verify whether the data source connection of the first data source is valid. The second method is used to verify whether the data source connection is valid; the second method may be, for example, `conn.isValid()`.

[0210] For example, the steps to obtain the data source connection of the first data source and verify the validity of the data source connection are as follows: Obtain the target object DataSourceParam from the utility library class; Define the main class DataSourceWithParamDemo; Define the main method, main(). Configure the target object, including configuring the following connection information: the address of the target server where the first data source is located and the port number of the port on the target server where the first data source provides services; the account for accessing the first data source; and the password for accessing the first data source. Optionally, the full path of the Java Database Connectivity (JDBC) driver class of the first data source can also be configured to the target object. Another optional configuration is to configure the following connection parameters to the target object: the maximum number of data source connections; the minimum number of idle data source connections; the timeout duration for acquiring a data source connection (milliseconds); the idle timeout duration for a data source connection (milliseconds); and the maximum lifespan of a data source connection (milliseconds). The first method, DataSourceUtil.create(), is called, and the target object containing the connection information of the first data source, the full path of the JDBC driver class of the first data source, and the connection parameters is passed as the input parameters to the first method to create an instance of the first data source. The second method, getConnection(), is called to obtain a data source connection conn from the connection pool, verify whether the data source connection conn is valid (1000 represents the timeout period in milliseconds), and print the verification result. Capture and print exceptions (such as connection timeout, parameter error, etc.) that occur during the process of obtaining or verifying the data source connection conn.

[0211] In one possible implementation, the first computing device can specifically obtain the data source connection of the first data source based on the connection information of the first data source by coordinating the cooperation of various components in the scheduling system. For example, it includes the following steps: S1201-S1202.

[0212] S1201, obtains the data source connection of the first data source based on the connection information of the first data source through the data source plugin.

[0213] In one implementation of obtaining the data source connection of the first data source based on the connection information of the first data source through a data source plugin, the data source connection of the first data source is obtained through a fourth component based on the connection information of the first data source.

[0214] Specifically, the fourth component performs the above-mentioned steps of obtaining the first data source instance and obtaining the data source connection of the first data source.

[0215] S1202, return the data source connection of the first data source to the task execution component.

[0216] In one implementation that returns the data source connection of a first data source to the task execution component, the data source connection of the first data source is returned to the task execution component, and then returned to the client. Based on the client, the data source connection of the first data source is returned to the task execution component via a channel.

[0217] In one implementation where the client returns the data source connection of the first data source to the task execution component via a channel, a fourth component passes the data source connection of the first data source to the client. The client then encapsulates the data source connection of the first data source to obtain the encapsulated data source connection, and returns the encapsulated data source connection of the first data source to the task execution component via the channel.

[0218] In this embodiment, the fourth component transmits the data source connection of the first data source, the client encapsulates the data source connection of the first data source and returns it to the task execution component via a channel. This allows the task execution component to access the first data source based on the encapsulated data source connection, achieving standardized transmission and secure encapsulation of the data source connection, as well as convenience and reliability for the task execution component to access the data source, thus strengthening the standardization of interaction between the plugin and the execution component. Furthermore, due to the independent processing logic of the aforementioned components, the original architecture and functionality of the scheduling system are not disrupted, ensuring that the original operating mechanism of the system remains unaffected.

[0219] S905, based on the data source connection of the first data source, accesses the first data source.

[0220] In one implementation of accessing a first data source based on a connection to that first data source, the first data source is accessed only if the verification result of the first data source connection is valid. Thus, by first verifying the validity of the first data source connection and then accessing the first data source based on a valid connection, access failures caused by invalid first data source connections can be avoided, improving the success rate and stability of first data source access and ensuring the reliability of the data interaction process.

[0221] In one implementation, the task execution component accesses the first data source based on the data source connection of the encapsulated first data source.

[0222] In one implementation, for a first task that is an SQL task, the task execution component accesses the first data source through a data source connection to execute the SQL statements in the first task execution instruction and receives the operation results (query result set, execution success / failure indicators, error codes, etc.) returned by the first data source. Afterward, the execution service node can write the operation results into the first task execution context and ultimately feed them back to the scheduler to update the task status (e.g., "execution successful" or "execution failed").

[0223] In some embodiments, the method further includes: disconnecting the data source connection of the first data source in response to completing the first task. This avoids the data source connection of the first data source from occupying system resources and network links for extended periods, reducing the consumption of idle data source connections in the connection pool.

[0224] In some embodiments, the method further includes: receiving a second task, the second task indicating access to a second data source, the second task including a type item for the second data source, the type item including connection information of the second data source; obtaining a data source connection of the second data source based on the connection information of the second data source, the data source connection of the second data source representing a communication channel between the first computing device and the second data source; and accessing the second data source based on the data source connection of the second data source.

[0225] In this embodiment, the data source connection of the second data source is obtained based on the connection information of the second data source. The second data source is then accessed, achieving targeted and reliable access to it. In one implementation, a second task execution instruction is generated based on the second task, and a second task execution context is created for the second task. The connection information of the second data source is stored in the second task execution context.

[0226] In one possible implementation, the seventh component in the data source plugin obtains the connection information of the second data source from the second task execution context. The fifth component in the data source plugin then creates a channel (i.e., the channel of the second data source) between the second data source and the task execution component based on this connection information. The sixth component is then created as a client for the second data source. The eighth component in the data source plugin obtains the data source connection of the second data source based on this connection information. This data source connection is then returned to the client of the second data source, which in turn returns the data source connection to the task execution component through the channel of the second data source, enabling the task execution component to execute the second task.

[0227] In one possible implementation, the specific implementation of obtaining the data source connection of the second data source based on the connection information of the second data source can be referred to the specific implementation of obtaining the data source connection of the first data source based on the connection information of the first data source, which will not be elaborated here.

[0228] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0229] like Figure 14 As shown, this application embodiment provides another computing device 500. The computing device 500 includes a processor 510 and a memory 520 for storing processor-executable instructions. When the processor 510 is configured to execute instructions, the computing device 500 implements the data source access method as described above.

[0230] Figure 14 The computing device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0231] The computing device 500 is presented in the form of a general-purpose computing device. The components of the computing device 500 may include, but are not limited to: one or more processors 510, memory 520, communication bus 540 connecting different system components (including memory 520 and processor 510), and communication interface 530.

[0232] The communication bus 540 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0233] Computing device 500 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device, including volatile and non-volatile media, removable and non-removable media.

[0234] Memory 520 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computing device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 14 As not shown, a disk drive may be provided for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and a removable non-volatile optical disk (e.g., a compact disc read-only memory; hereinafter referred to as CD). ROM (Read-Only Memory Disc); Digital Video Disc Read Only Memory (hereinafter referred to as DVD) An optical disc drive that reads and writes to ROM or other optical media. In these cases, each drive can be connected to the communication bus 540 through one or more data media interfaces. The memory 520 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0235] A program / utility having a set (at least one) of program modules can be stored in memory 520. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0236] The computing device 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the computing device, and / or with any device that enables the computing device to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed through the communication interface 530. Furthermore, the computing device 500 can also communicate through a network adapter (… Figure 14 (Not shown) communicates with one or more networks (e.g., Local Area Network (LAN), Wide Area Network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the computing device via the communication bus 540. It should be understood that, although... Figure 14 As not shown, the computing device 500 may be used with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Drives (RAID) systems, tape drives, and data backup storage systems.

[0237] The processor 510 executes various functional applications and data processing by running programs stored in the memory 520, such as implementing the methods described above in the embodiments of this application.

[0238] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the computing device 500. In other embodiments of this application, the computing device 500 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0239] It is understood that the aforementioned computing devices, etc., include hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the exemplary units and algorithm steps described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0240] This application embodiment can divide the above-mentioned computing device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0241] This application also provides a storage medium storing computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method described above.

[0242] This application also provides a computer program product, which includes a computer program that, when at least one processor executes the computer program, causes the at least one processor to perform the methods described above in this application.

[0243] The computing device, storage medium, or computer program product provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0244] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0245] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0246] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0247] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for accessing a data source, characterized in that, Applied to a first computing device; the method includes: Receive a first task, the first task being used to instruct access to a first data source, the first task including a type item of the first data source, the type item of the first data source including identification information of the first data source, the identification information of the first data source being used by the first computing device to obtain connection information of the first data source from the second computing device; Based on the identification information of the first data source, the connection information of the first data source is obtained from the second computing device; Based on the connection information of the first data source, the data source connection of the first data source is obtained, and the data source connection of the first data source is used to represent the communication channel between the first computing device and the first data source. Access the first data source based on the data source connection of the first data source.

2. The method according to claim 1, characterized in that, The step of obtaining the data source connection of the first data source based on the connection information of the first data source includes: Based on the connection information of the first data source, a first data source instance is obtained, and the first data source instance is used to obtain the data source connection of the first data source; Invoke the first data source instance to obtain the data source connection of the first data source.

3. The method according to claim 2, characterized in that, The step of obtaining the first data source instance based on the connection information of the first data source includes: Obtain the target object and configure the connection information of the first data source into the target object, wherein the target object is used to encapsulate the connection information of the first data source; The first method is invoked, and the target object configured with the connection information of the first data source is used as the input parameter of the first method to create the first data source instance. The first method is used to create the first data source instance. Invoke the first data source instance to obtain the data source connection of the first data source, including: Invoke the first data source instance to create a data source connection for the first data source.

4. The method according to claim 2, characterized in that, The step of obtaining the first data source instance based on the connection information of the first data source includes: Based on the connection information of the first data source, a data source instance corresponding to the connection information of the first data source is determined from the data source instance set and used as the first data source instance. The data source instance set includes data source instances corresponding to the connection information of different data sources. The first data source instance includes multiple data source connections. The connection parameters of the multiple data source connections are different. The connection parameters include one or more of the following: number of connections, connection status, and lifecycle. The step of calling the first data source instance to obtain the data source connection of the first data source includes: The first data source instance is invoked to determine the data source connection of the first data source from the plurality of data source connections based on the connection parameters.

5. The method according to any one of claims 2-4, characterized in that, Before accessing the first data source via the data source connection based on the first data source, the method further includes: The second method is invoked to verify whether the data source connection of the first data source is valid. The second method is used to verify whether the data source connection is valid. The data source connection based on the first data source, accessing the first data source, includes: If the data source connection of the first data source is valid, the first data source is accessed based on the data source connection of the first data source.

6. The method according to any one of claims 1-5, characterized in that, Prior to receiving the first task, the method further includes: In response to the user's selection of the type of the first data source, a configuration interface is displayed. The configuration interface includes a first option, which is used by the user to configure the identification information of the first data source. Receive the identification information of the first data source configured by the user in the first option.

7. The method according to any one of claims 1-6, characterized in that, Before obtaining the connection information of the first data source from the second computing device based on the identification information of the first data source, the method further includes: The interface configuration information of the second computing device is obtained from the preset configuration file. The interface configuration information includes the interface address and interaction method of the second computing device. The step of obtaining the connection information of the first data source from the second computing device based on the identification information of the first data source includes: Based on the identification information of the first data source and the interface configuration information of the second computing device, the connection information of the first data source is obtained from the second computing device.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Receive a second task, the second task being used to instruct access to a second data source, the second task including a type item of the second data source, the type item of the second data source including connection information of the second data source; Based on the connection information of the second data source, the data source connection of the second data source is obtained, and the data source connection of the second data source is used to represent the communication channel between the first computing device and the second data source; Access the second data source based on the data source connection of the second data source.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: In response to the completion of the first task, the data source connection of the first data source is disconnected.

10. A computing device, characterized in that, It includes a processor and a memory; the processor is coupled to the memory; The memory is used to store instructions; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-9.

Images