Data access method and device based on multiple data sources, equipment and storage medium

By generating declarative interfaces and query plan information, data operation plans are automatically generated and pushed down to the data source for execution, solving the problem of low data access efficiency in the lake warehouse integrated architecture and achieving efficient data access and flexible database operations.

CN121880451APending Publication Date: 2026-04-17RICHFIT INFORMATION TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
RICHFIT INFORMATION TECH
Filing Date
2025-10-15
Publication Date
2026-04-17

AI Technical Summary

Technical Problem

In a lake-warehouse integrated architecture, the difference in storage media between the data lake and the data warehouse leads to low data access efficiency, making it difficult to meet the needs of efficient data analysis such as machine learning.

Method used

By generating declarative interface information and query plan information, a data operation plan is automatically generated and pushed down to the data source for execution, avoiding the return of a large amount of raw data.

Benefits of technology

It improves data access efficiency, enhances the flexibility of upper-layer applications, reduces the need for dynamic configuration of different storage media syntaxes, and improves the efficiency of user database operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121880451A_ABST
    Figure CN121880451A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data access method and device based on multiple data sources, equipment and a storage medium. The method comprises the following steps: receiving data attribute information sent by a user; generating declarative interface information according to the data attribute information; wherein the declarative interface information is a declarative data processing interface and represents data to be accessed by a user; generating query plan information according to the declarative interface information; wherein the query plan information comprises data operation to be executed when the to-be-accessed data is accessed; and pushing down the data operation in the query plan information to a data source of the to-be-accessed data to obtain the to-be-accessed data. The method is used for improving the data access efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data access method, apparatus, device and storage medium based on multiple data sources. Background Technology

[0002] In enterprise data management, a lake warehouse architecture is used to manage the data generated by the enterprise during its business processes.

[0003] In a lake-warehouse architecture, the data lake and data warehouse each emphasize different functions and characteristics, and the storage media they provide also differ significantly. The storage media provided by the data lake prioritizes cost-effectiveness and flexible scalability, while the storage media provided by the data warehouse prioritizes performance and reliability. The database comprised of the data lake and data warehouse presents challenges to efficient data access. Therefore, improving data access efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a data access method, apparatus, device, and storage medium based on multiple data sources to improve data access efficiency.

[0005] In a first aspect, embodiments of this application provide a data access method based on multiple data sources, including:

[0006] Receive data attribute information sent by the user;

[0007] Based on the data attribute information, declarative interface information is generated; wherein, the declarative interface information is a declarative data processing interface that represents the data that the user wants to access;

[0008] Based on the declarative interface information, query plan information is generated; wherein, the query plan information includes the data operations to be performed when accessing the data to be accessed;

[0009] The data operations in the query plan information are pushed down to the data source of the data to be accessed, thus obtaining the data to be accessed.

[0010] In one possible implementation, the data attribute information includes the data source type of the data to be accessed; based on the data attribute information, declarative interface information is generated, including:

[0011] Based on the data source type, determine the data source connection method corresponding to the data source type; wherein, the data source connection method represents the way to connect to the database in the data source;

[0012] Based on the data attribute information and the data source connection method corresponding to the data source type, the declarative interface information is generated based on a preset operation function.

[0013] In one possible implementation, generating the declarative interface information based on the data attribute information and the data source connection method corresponding to the data source type, and based on a preset operation function, includes:

[0014] Based on the data attribute information and the data source connection method corresponding to the data source type, task flow orchestration information is generated based on preset operation functions; wherein, the task flow orchestration information represents the dependencies and execution order between data operations;

[0015] Based on the task flow orchestration information, the declarative interface information is generated.

[0016] In one possible implementation, generating query plan information based on the declarative interface information includes:

[0017] The declarative interface information is parsed to construct an abstract syntax tree; wherein, the abstract syntax tree represents the declarative interface information, and the nodes in the abstract syntax tree represent data operations;

[0018] The query plan information is generated based on the abstract syntax tree.

[0019] In one possible implementation, parsing the declarative interface information and constructing an abstract syntax tree includes:

[0020] Lexical analysis is performed on the declarative interface information to decompose it into at least one lexical unit.

[0021] Syntactic analysis is performed on each lexical unit to construct the abstract syntax tree.

[0022] In one possible implementation, generating the query plan information based on the abstract syntax tree includes:

[0023] The nodes in the abstract syntax tree are divided to obtain at least one combination of nodes;

[0024] For each node combination, a query statement fragment is generated; wherein, the query statement fragment represents the data operation in the node combination;

[0025] Based on the order of nodes in the abstract syntax tree, the query statement fragments are combined into the query plan information.

[0026] In one possible implementation, the step of pushing down the data operation in the query plan information to the data source to be accessed, thereby obtaining the data to be accessed, includes:

[0027] Determine the storage medium where the data source of the data to be accessed is located, and determine the target language type corresponding to the storage medium; wherein, the target language type represents the type of query language supported by the storage medium;

[0028] Based on the target language type, the data operations in the query plan information are converted into the target query statement;

[0029] Based on a preset query engine, the target query statement is pushed down to the data source of the data to be accessed, thereby obtaining the data to be accessed.

[0030] In one possible implementation, the step of converting the data operations in the query plan information into a target query statement according to the target language type includes:

[0031] Identify a first operation and a second operation from the query plan information; wherein the first operation represents a data operation that allows pushdown, and the second operation represents a data operation that does not allow pushdown;

[0032] Based on the target language type, the first operation in the query plan information is converted into the target query statement;

[0033] The second operation in the query plan information is retained in the preset query engine.

[0034] Secondly, embodiments of this application provide a data access device based on multiple data sources, including:

[0035] The receiving module is used to receive data attribute information sent by the user;

[0036] The first generation module is used to generate declarative interface information based on the data attribute information; wherein, the declarative interface information is a declarative data processing interface that represents the data that the user wants to access;

[0037] The second generation module is used to generate query plan information based on the declarative interface information; wherein, the query plan information includes the data operations to be performed when accessing the data to be accessed;

[0038] The push-down module is used to push down the data operations in the query plan information to the data source of the data to be accessed, so as to obtain the data to be accessed.

[0039] Thirdly, embodiments of this application provide a data access device based on multiple data sources, including: a memory and a processor;

[0040] The memory stores computer-executed instructions;

[0041] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0043] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0044] The data access method, apparatus, device, and storage medium based on multiple data sources provided in this application receive data attribute information from a user and generate declarative interface information based on the data attribute information. The declarative interface information is a declarative data processing interface that represents the data the user wants to access. By generating declarative interface information, users do not need to concern themselves with the storage differences between underlying data sources; they only need to define the data operation flow, enhancing the flexibility of upper-layer applications. After generating the declarative interface information, query plan information is generated based on it. The query plan information includes the data operations to be performed when accessing the data to be accessed. By automatically generating the query plan, users do not need to dynamically configure the syntax of different storage media, improving the efficiency of user database operations. After generating the query plan information, the data operations in the query plan information are pushed down to the data source of the data to be accessed, obtaining the data to be accessed. Through the push-down operation, data operations are directly executed on the data source, avoiding the return of a large amount of raw data; only a small amount of data operations are returned after completion, thereby improving data access efficiency. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] Figure 1 A flowchart illustrating the data access method based on multiple data sources provided in this application;

[0047] Figure 2 The flowchart for generating declarative interface information provided in this application Figure 1 ;

[0048] Figure 3 The flowchart for generating declarative interface information provided in this application Figure 2 ;

[0049] Figure 4 The flowchart for generating query plan information provided in this application Figure 1 ;

[0050] Figure 5 A flowchart illustrating the process of constructing an abstract syntax tree provided in this application;

[0051] Figure 6 The flowchart for generating query plan information provided in this application Figure 2 ;

[0052] Figure 7 A flowchart illustrating the pushdown data operation provided in this application;

[0053] Figure 8 A flowchart illustrating the process of converting data operations in query plan information into target query statements, provided for this application;

[0054] Figure 9 A schematic diagram of the structure of the data access device based on multiple data sources provided in this application;

[0055] Figure 10 This is a schematic diagram of the structure of the data access device based on multiple data sources provided in this application.

[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0058] First, let me explain the terms used in this application:

[0059] Data lake: A centralized storage system used to centrally store raw, unprocessed data;

[0060] A data warehouse is a highly structured, integrated data storage system used to store data that has undergone processing, cleaning, modeling, and other data processing operations.

[0061] A DataFrame is a two-dimensional tabular data structure represented in memory with row and column labels. It is the primary object of operation for data cleaning, analysis, and machine learning. It provides a series of high-level methods (such as `filter`, `groupBy`, and `join` functions) to describe operations on the database, rather than step-by-step instructions.

[0062] SQL stands for Structured Query Language. It is a domain-specific language specifically designed for managing and manipulating relational databases.

[0063] In the era of big data and artificial intelligence, the number of enterprises is constantly increasing, and the sources of these enterprises are also diversifying, leading to a growing demand for advanced analytics such as machine learning. More and more companies are choosing to store their data in low-cost data lakes, and then connect this data to data warehouses for computation and analysis, forming a lake-warehouse integrated architecture that separates storage and computation. In this model, the data lake provides flexible, multi-type data storage capabilities, while the data warehouse provides fast and efficient computation, analysis, and data management capabilities. The two are integrated to jointly support the data access needs of enterprises for advanced analytics.

[0064] Data lakes and data warehouses emphasize different functions and characteristics, and therefore offer different storage media. Data lakes prioritize cost-effectiveness and flexible scalability, making them suitable for storing large volumes of diverse data, such as object storage, cloud storage, and distributed file systems. Data warehouses, on the other hand, prioritize performance and reliability, making them suitable for storing medium-sized structured data and offering better support for complex data analysis and queries, such as relational databases, columnar storage databases, and in-memory databases.

[0065] In a lake-warehouse integrated architecture, the database comprised of the data lake and the data warehouse presents challenges for efficient data access. For example, the data warehouse provides data via JDBC (Java Database Connectivity, a standard Java interface defining how Java programs interact with various databases), which is suitable for retrieving small amounts of data. However, for high-efficiency data analysis needs such as machine learning, large datasets need to be retrieved from the database at once. When retrieving large datasets at once, data transfer efficiency is significantly reduced. But if data analysis is performed directly on the data lake, the transaction management features of the data warehouse are lost, potentially leading to data errors during data transfer between storage and computing engines.

[0066] The data access method, apparatus, device, and storage medium based on multiple data sources provided in this application are intended to solve the aforementioned technical problems.

[0067] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0068] Figure 1 A flowchart illustrating the data access method based on multiple data sources provided in this application is shown below. Figure 1 As shown, the method includes:

[0069] S101, Receive data attribute information sent by the user.

[0070] It should be noted that the subject of this application may be an electronic device or apparatus with data processing capabilities.

[0071] In a lakeware environment, data may be stored in databases of different data source types, including relational databases, columnar storage databases, in-memory databases, distributed file systems, and object storage, each corresponding to a different storage system. Therefore, different data sources require different connection methods.

[0072] For example, the system receives data attribute information from the user. This data attribute information represents the data required to connect to the database. For instance, the attribute information includes the data source type and the configuration parameters required to connect to the data source.

[0073] For example, a data source type selection list can be provided, with items including relational databases, columnar storage databases, in-memory databases, distributed file systems, and object storage. The user selects at least one target data source type from the list. For each selected target data source type, the system automatically matches a database connection driver and prompts the user to enter the required configuration parameters for each target data source type. The database connection driver is used to connect to the database. Configuration parameters include the data source connection string (such as the data source address information, port information, etc.) and the storage path of the data tables in the data source. Different data sources can have different configuration parameters.

[0074] S102. Generate declarative interface information based on data attribute information; wherein, the declarative interface information is a declarative data processing interface that represents the data that the user wants to access.

[0075] For example, based on data attribute information, the data source is determined. There may be one or more data sources, and each data source corresponds to a data table in a database of a specific data source type. For instance, based on the data source type in the data attributes, the data source is determined to be table A in a relational database. Declarative interface information can be generated based on the data source. For example, after determining the data source, declarative interface information is generated based on the data operations performed on the data from the data source as needed. There may be one or more data operations performed on the data from the data source. These data operations are user-specified operations, including the data being operated on and the type of operation (e.g., selection, join, aggregation, etc.). The declarative interface information is a declarative data processing interface that represents the data the user wants to access. The declarative interface information includes one or more data operations corresponding to each data source.

[0076] For example, based on a DataFrame, data operations on the data in table A can be defined as functions. Each function corresponds to one data operation on the database, and the parameters of each function correspond to data in the database. Since one data source corresponds to one or more data operations, there are one or more data operations performed on the data in table A. The declarative interface information corresponding to table A corresponds to one or more functions, and these one or more functions together constitute the declarative interface information.

[0077] S103. Generate query plan information based on the declarative interface information; wherein, the query plan information includes the data operations to be performed when accessing the data to be accessed.

[0078] For example, query plan information, such as SQL-based query plan information, can be generated based on declarative interface information. This query plan information includes the data operations to be performed when accessing the data to be accessed. One declarative interface information can generate one query plan information, which includes one or more data operations.

[0079] For example, lexical analysis tools can be used to perform lexical analysis on one or more functions in the declarative interface information, resulting in multiple lexical units, such as function names, parameters, and operators. These lexical units represent the smallest unit of information in the declarative interface information.

[0080] Then, based on preset syntax analysis rules, the grammatical relationships between one or more lexical units are analyzed. One or more lexical units that satisfy the preset syntax analysis rules are taken as a syntax analysis result. The preset syntax analysis rules are used to analyze the grammatical relationships between one or more lexical units.

[0081] Then, based on a preset mapping relationship, each syntax analysis result is mapped to a corresponding query statement fragment. The preset mapping relationship is used to map the syntax analysis results to query statement fragments.

[0082] Finally, based on preset combination rules, all query statement fragments are combined to obtain query plan information. The preset combination rules are used to combine query statement fragments to obtain query plan information.

[0083] S104. Push down the data operation in the query plan information to the data source of the data to be accessed, and obtain the data to be accessed.

[0084] For example, after obtaining the query plan, the data operations in the query plan are pushed down to the data source of the data to be accessed, and the data source engine performs the data operations to obtain the accessed data.

[0085] For example, for any data operation in the query plan, based on preset pushdown rules, it is determined whether the data operation can be pushed down to the data source. The preset pushdown rules are used to determine whether a data operation can be pushed down to the data source. If it can be pushed down to the data source, the scheduling task of the data operation is pushed down to the engine of the data source to which it is pushed down; if it cannot be pushed down, the scheduling task of the data operation is retained in the engine of the calling end.

[0086] Each data operation's corresponding engine executes its respective data operation and returns the data retrieved from the database. The data returned from the database by all data operations represents the accessed data.

[0087] The data access method based on multiple data sources provided in this application receives data attribute information from a user and generates declarative interface information based on this information. The declarative interface information is a declarative data processing interface that represents the data the user wants to access. By generating declarative interface information, users do not need to concern themselves with the storage differences between underlying data sources; they only need to define the data operation flow, enhancing the flexibility of upper-layer applications. After generating the declarative interface information, query plan information is generated based on it. The query plan information includes the data operations to be performed when accessing the data to be accessed. By automatically generating the query plan, users do not need to dynamically configure the syntax of different storage media, improving the efficiency of user database operations. After generating the query plan information, the data operations in the query plan information are pushed down to the data source of the data to be accessed, obtaining the data to be accessed. Through the pushdown operation, data operations are directly executed on the data source, avoiding the return of a large amount of raw data; only a small amount of data operations are returned after completion, thereby improving data access efficiency.

[0088] The following uses the Enterprise Resource Management System (ERP) of a petroleum company as an example to further illustrate the technical solution of this application. The ERP of this petroleum company covers multiple core business modules, including production, procurement, sales, and finance. The company's massive historical operational data (such as equipment operation logs and raw material consumption records) is stored in a distributed file system. Customer information and sales order data are stored in a relational database. Some financial voucher data is stored in an object storage database. The petroleum company hopes to leverage the data from the ERP to analyze the production costs, sales profits, and customer purchasing behavior of different product lines, in order to optimize production resource allocation, formulate marketing strategies, and improve the accuracy of financial decisions.

[0089] Figure 2 The flowchart for generating declarative interface information provided in this application Figure 1 The data attribute information includes the data source type of the data to be accessed, such as... Figure 2 As shown, in step S102 above, generating declarative interface information based on data attribute information includes the following sub-steps:

[0090] S201. Determine the data source connection method corresponding to the data source type based on the data source type; wherein, the data source connection method represents the way to connect to the database in the data source.

[0091] For example, the data attribute information includes the data source type of the data to be accessed. For instance, the data attribute information might include a distributed file system, a relational database, or object storage, selected by the user from a list of data source types.

[0092] For each data source type, the corresponding data source connection method is determined based on the data source type. The data source connection method characterizes the way the connection is established with the database within the data source. The data source connection method corresponding to the data source type includes: the driver for connecting to the data source, and the configuration parameters required for connecting to the data source. Configuration parameters include the data source connection string (such as the data source address information, port information, etc.) and the storage path of the data tables within the data source.

[0093] For example, for a selected distributed file system, based on preset driver matching rules, the connection method for the distributed file system is determined, including using a JDBC driver to connect to the database. The preset driver matching rules are used to determine the database driver corresponding to different data sources. The database driver is used to connect to the database. Simultaneously, the configuration parameters prompted to the user include: the database connection address, port information, and the storage path of the data tables. The system obtains the user-inputted information such as the distributed file system connection address, port information, and data table storage path. Using a JDBC driver to connect to the database, along with the obtained user-inputted information such as the distributed file system connection address, port information, and data table storage path, determines the connection method for the distributed file system.

[0094] S202. Based on the data attribute information and the data source connection method corresponding to the data source type, generate declarative interface information based on the preset operation function.

[0095] For example, after determining the data source connection method corresponding to the data source type, declarative interface information is generated based on the data attribute information and the data source connection method corresponding to the data source type, and on the basis of the preset operation function.

[0096] For example, there are multiple preset operation functions, which represent the data processing steps. These preset operation functions include filter functions for data filtering, map functions for data mapping, join functions for data joining, and reduce functions for data aggregation. Any one or more preset operation functions can be used to generate declarative interface information.

[0097] Data source information can be generated based on the data source driver in the data source connection method corresponding to the data source type, as well as the data table storage path information. Data attribute information also includes the data to be accessed and the data operations performed on that data. Based on the data to be accessed and the data operations performed on that data, combined with preset operation functions, data processing information is generated. Declarative interface information includes data source information and data processing information.

[0098] For example, declarative interface information includes: using JDBC drivers to read data from historical operation data tables in a distributed file system, customer information tables in a relational database, sales order tables in a relational database, and financial voucher data tables in an object storage database. It also includes using filter functions to filter data for specific time periods, such as filtering records where equipment operation hours exceeded 1000 hours and raw material consumption exceeded 10,000 yuan in the past year. Map functions are used to extract key information, such as extracting equipment numbers and operation information from equipment logs. Customer and order data are read from the relational database, and join functions are used to connect the corresponding order information for each customer. The data processed by the distributed file system and relational database are joined by product line number, and then joined with the financial data in the object storage database by order number. Finally, reduce functions are used to aggregate the data, calculate the cost and profit for each product line, and sort the data.

[0099] In this embodiment, the data attribute information includes the data source type of the data to be accessed. Based on the data source type, the corresponding data source connection method is determined. Then, based on the data attributes and the data source connection method corresponding to the data source type, declarative interface information is generated using a preset operation function. The generated declarative interface information includes database connection information, facilitating subsequent connection to the corresponding database. Defining operations on data in the data source through functions allows developers to manipulate data in the database using custom functions. Compared to directly defining query statements, the functional approach is simpler and more efficient.

[0100] Figure 3 The flowchart for generating declarative interface information provided in this application Figure 2 ,like Figure 3 As shown, in step S202 above, based on data attribute information and the data source connection method corresponding to the data source type, declarative interface information is generated based on a preset operation function, including the following sub-steps:

[0101] S301. Based on the data attribute information and the data source connection method corresponding to the data source type, generate task flow orchestration information based on the preset operation function; wherein, the task flow orchestration information represents the dependency relationship and execution order between data operations.

[0102] For example, based on data attribute information and the data source connection method corresponding to the data source type, task flow orchestration information is generated based on preset operation functions. The task flow orchestration information represents the dependencies and execution order between data operations.

[0103] For example, in the declarative interface information, data is read from historical execution data tables in a distributed file system, customer information tables in a relational database, sales order tables in a relational database, and financial voucher data tables in an object storage database using a JDBC driver, which is the first data operation in the task flow orchestration information.

[0104] Then, based on the execution order of the functions, the dependencies and execution order of the data operations corresponding to each function are generated. For example, the `filter` function is used to filter data for a specific time period, such as records where equipment has run for more than 1000 hours and raw material consumption has exceeded 10,000 yuan in the past year, which is the second data operation. The `map` function is used to extract key information, such as extracting equipment number and operation information from equipment logs, which is the third data operation. Customer and order data are read from the relational database, and the `join` function is used to join the order information corresponding to the customer, which is the fourth data operation. The data processed by the distributed file system and the relational database is joined by product line number, and then joined with the financial data in the object storage database by order number, which is the fifth data operation. Finally, the `reduce` function is used to aggregate the data, calculate the cost and profit of each product line, and sort them, which is the sixth data operation.

[0105] For any two adjacent data operations among multiple data operations in the task flow orchestration information, the latter data operation can obtain the data processing result of the former data operation, and the obtained data processing result of the former data operation can be used as the input data for the latter data operation.

[0106] S302. Generate declarative interface information based on task flow orchestration information.

[0107] For example, declarative interface information is generated based on task flow orchestration information. For instance, the content of the declarative interface information includes task flow orchestration information.

[0108] In this embodiment, task flow orchestration information is generated based on data attribute information and the data source connection method corresponding to the data source type, and a preset operation function. Furthermore, declarative interface information is generated based on the task flow orchestration information. This enables the definition of multiple chained data operations through declarative interface information, avoiding manual management of data transfer between different operations, thereby reducing the workload of writing complex low-level code and improving database utilization efficiency.

[0109] Figure 4 The flowchart for generating query plan information provided in this application Figure 1 ,like Figure 4 As shown, in step S103 above, the query plan information is generated based on the declarative interface information, including the following sub-steps:

[0110] S401. Parse the declarative interface information and construct an abstract syntax tree; wherein, the abstract syntax tree represents the declarative interface information, and the nodes in the abstract syntax tree represent data operations.

[0111] For example, the declarative interface information is parsed to construct an abstract syntax tree. The abstract syntax tree represents the declarative interface information, and the nodes within the abstract syntax tree represent data operations.

[0112] For example, lexical analysis tools can be used to perform lexical analysis on one or more functions in declarative interface information, resulting in multiple lexical units, such as function names, parameters, and operators. These lexical units represent the smallest unit of information in the declarative interface information. For instance, the function `filter`, which filters data for a specific time period, such as records where equipment has run for more than 1000 hours and raw material consumption has exceeded 10,000 yuan in the past year, can be decomposed into 'filter', 'year=2024', 'runTime>1000', and 'cost>10000'.

[0113] Then, based on preset syntax analysis rules, the grammatical relationships between one or more lexical units are analyzed. One or more lexical units that satisfy the preset syntax analysis rules are taken as a syntax analysis result. The preset syntax analysis rules are used to analyze the grammatical relationships between one or more lexical units, and these grammatical relationships are the data operations formed by those lexical units. Based on the grammatical relationships between the syntax analysis results, an abstract syntax tree is generated, where each node of the abstract syntax tree represents a syntax analysis result. The connections between nodes in the abstract syntax tree represent the connections between data operations.

[0114] S402. Generate query plan information based on the abstract syntax tree.

[0115] For example, query plan information is generated based on the abstract syntax tree (API). For instance, starting from the root node, each node of the API is traversed, and the connections between nodes are analyzed. By analyzing these connections, the chained call relationships between multiple data operations can be obtained. Based on a predefined mapping relationship, the query plan information is obtained by converting data operations into corresponding query statement fragments and the chained call relationships between data operations into corresponding query statement fragments, and then concatenating all the query statement fragments.

[0116] In this embodiment, declarative interface information is parsed to construct an abstract syntax tree. Query plan information is then generated based on the abstract syntax tree. By parsing declarative interface information into a structured abstract syntax tree, the syntactic structure and logical intent of the user code can be accurately understood. Linear, user-readable code is transformed into tree-like, computer-processable structured data. This method facilitates the traversal, analysis, and manipulation of code logic, laying the foundation for subsequent optimization and code generation.

[0117] Figure 5 The flowchart illustrating the construction of the abstract syntax tree provided in this application is as follows: Figure 5 As shown, the process of parsing declarative interface information and constructing an abstract syntax tree in S402 includes the following sub-steps:

[0118] S501. Perform lexical analysis on the declarative interface information and decompose the declarative interface information into at least one lexical unit.

[0119] For example, lexical analysis is performed on declarative interface information to decompose it into at least one lexical unit. For instance, a lexical analysis tool can be used to perform lexical analysis on one or more functions within the declarative interface information, resulting in multiple lexical units, such as function names, parameters, and operators. Each lexical unit represents the smallest unit of information within the declarative interface information. For example, the `filter` function, which filters data for a specific time period, such as records where the equipment has run for more than 1000 hours and raw material consumption has exceeded 10,000 yuan in the past year, can be decomposed into 'filter', 'year=2024', 'runTime>1000', and 'cost>10000'.

[0120] S502. Perform syntactic analysis on each lexical unit to construct an abstract syntax tree.

[0121] For example, each lexical unit is subjected to syntactic analysis to construct an abstract syntax tree. For instance, based on a pre-defined syntactic analysis tool, the lexical units are combined into an abstract syntax tree.

[0122] In this embodiment, lexical analysis is performed on declarative interface information, decomposing it into at least one lexical unit. This achieves precise lexical unit segmentation and recognition, breaking down a continuous character stream into meaningful, discrete lexical units, eliminating characters such as spaces and comments that are useless for logical understanding, and preparing for subsequent syntax analysis. After lexical analysis, syntax analysis is performed on each lexical unit to construct an abstract syntax tree. By constructing the abstract syntax tree, the rationality of the lexical units is checked, syntax errors can be detected and reported, improving the accuracy of subsequent data processing. Furthermore, the constructed abstract syntax tree directly reflects the semantics of the code, facilitating subsequent machine recognition of the priority and dependencies of data operations.

[0123] Figure 6 The flowchart for generating query plan information provided in this application Figure 2 ,like Figure 6 As shown, in step S402 above, generating query plan information based on the abstract syntax tree includes the following sub-steps:

[0124] S601. Divide the nodes in the abstract syntax tree to obtain at least one node combination.

[0125] For example, the nodes in the abstract syntax tree are partitioned to obtain at least one combination of nodes. For instance, starting from the root node, all nodes in the abstract syntax tree are traversed and analyzed to identify the data operations contained in the nodes.

[0126] Common data operations include selection, join, aggregation, sorting, and deduplication. Since data operations in an abstract syntax tree typically appear as predefined function calls—for example, the `filter` function filters data, the `map` function maps data, and the `reduce` function aggregates data—common operation functions can be mapped to their corresponding data operations to form a predefined semantic understanding library.

[0127] When traversing and analyzing the abstract syntax tree, this semantic understanding library enables fast lookup and matching, identifying a node, or a node and one or more adjacent nodes constituting a data operation. A node, or a node and one or more adjacent nodes, is grouped into a node combination. For example, the parameters of a binary operation node and the binary operations in its left and right subtrees are grouped into a combination to generate a conditional expression of the data operation.

[0128] After generating node combinations using a semantic understanding library, for identified data operations, the system can determine the data operation objects (such as databases, tables, fields, etc.) based on the parameters of preset operation functions. Then, it can parse the data source of each data object to determine which data source in the data lake or data warehouse the data originates from. Simultaneously, it analyzes the dependencies and execution order between data operations, constructing an abstract representation of the data preparation process using tools such as directed acyclic graphs (DAGs). Each data operation is viewed as a first node in a DAG, and directed edges between these first nodes represent the dependencies between them. For example, when selecting data for sorting, two first nodes in DAGs can be created to represent the selection and sorting operations, respectively. A directed edge is added between these two first nodes to indicate that the sorting operation depends on the result of the selection operation. Using graph theory algorithms such as topological sorting to analyze the abstract representation in data preparation, the execution order of data operations can be determined, providing a basis for subsequently constructing a query plan.

[0129] S602. For each node combination, generate a query statement fragment; wherein, the query statement fragment represents the data operation in the node combination.

[0130] For example, for each node combination, a query statement fragment is generated. This query statement fragment represents the data operations within the node combination. For instance, the data operations of each identified node combination are converted into query statement fragments for the corresponding data objects. For example, selection operations are converted into SELECT clauses in SQL-based query languages, join operations into JOIN clauses, aggregation operations into GROUP BY clauses or aggregate functions, and sorting operations into ORDER BY clauses, etc.

[0131] S603. Based on the order of nodes in the abstract syntax tree, combine the query statement fragments into query plan information.

[0132] For example, query statement fragments are combined into query plan information according to the order of nodes in the abstract syntax tree. For instance, query statement fragments are combined according to the order of nodes in the data abstract syntax tree, and during the combination process, logical operators such as AND (AND logical operation) and OR (OR logical operation) from the query language are added to ensure the syntax and logic of the query statements are correct, thus generating complete query plan information.

[0133] In this embodiment, the nodes in the abstract syntax tree are divided to obtain at least one node combination. Strategically decomposing a large abstract syntax tree into logically related node combinations breaks down the complex translation task into a series of simple, manageable node combinations, providing the possibility for subsequent fine-grained optimization for specific types of node combinations. For each node combination, query statement fragments are generated, and according to the order of nodes in the abstract syntax tree, these query statement fragments are combined into query plan information. This establishes a precise mapping from the abstract syntax tree to the query plan information, ensuring that every operation in the declarative interface information is converted into equivalent query semantics without omission or ambiguity, thereby guaranteeing the accuracy of the generated query plan information.

[0134] Figure 7 The flowchart illustrating the pushdown data operation provided in this application is as follows: Figure 7 As shown, in S104 above, the data operation in the query plan information is pushed down to the data source of the data to be accessed to obtain the data to be accessed, including the following sub-steps:

[0135] S701. Determine the storage medium where the data source of the data to be accessed is located, and determine the target language type corresponding to the storage medium; wherein, the target language type represents the type of query language supported by the storage medium.

[0136] For example, the storage medium containing the data source of the data to be accessed is determined, and the target language type corresponding to the storage medium is determined. Here, the target language type represents the type of query language supported by the storage medium. The data to be accessed is the data operated on by any data operation.

[0137] It's important to note that the database maintains a metadata directory, which allows access to the database and its data's metadata. For example, it can retrieve data tables and the storage paths of the data within them. Therefore, the storage path of the data source for the data to be accessed can be obtained from the metadata directory. Based on this storage path (which includes storage medium information), the storage medium where the data source resides can be determined. Then, based on a predefined mapping between storage media and query language types, the target language type corresponding to the storage medium can be determined.

[0138] S702. Based on the target language type, convert the data operations in the query plan information into the target query statement.

[0139] For example, based on the target language type, the data operations in the query plan information are converted into the target query statement. For instance, based on preset language conversion rules, the data operations in the query plan information can be converted into the target query statement according to the target language type. The preset language conversion rules are used to convert the data operations in the query plan into the target query statement in the target language type, and these preset language conversion rules are formulated based on the function library and syntax of the target language type.

[0140] S703. Based on the preset engine, the target query statement is pushed down to the data source of the data to be accessed, and the data to be accessed is obtained.

[0141] For example, based on a preset query engine, the target query statement is pushed down to the data source of the data to be accessed, thus obtaining the data to be accessed. For instance, through the interface between the preset query engine (such as an SQL engine) and the storage location of the data to be accessed, the target query statement is sent to the data source to be accessed; that is, the target query statement is sent to the storage location of the data to be accessed for execution. By executing the target query statement, the data to be accessed is obtained.

[0142] In this embodiment, the storage medium where the data source of the data to be accessed is located is determined, and the target language type corresponding to the storage medium is also determined. This ensures that the pushdown operation can be understood and executed optimally by the data source engine. Based on the target language type, the data operations in the query plan information are converted into a target query statement. The generated target query statement fully conforms to the syntax specifications of the pushdown engine, guaranteeing the executability of the pushdown operation. Furthermore, it can utilize the optimizer built into the data source to achieve optimal performance. Thus, based on the preset query engine, the target query statement is pushed down to the data source of the data to be accessed, obtaining the data to be accessed. This achieves stable and correct execution of the pushdown operation.

[0143] In some specific implementations, when the target query statement is pushed down to the data source of the data to be accessed for execution, the database performance metrics, such as query execution time, data processing speed, and resource usage, can be monitored in real time during the execution of the target query statement. If query performance issues are detected, such as excessively long execution time or high resource consumption, the execution parameters of the target query statement (such as query concurrency, filtering conditions, and data read block size) should be adjusted promptly to ensure that the target query statement can be executed efficiently.

[0144] In some specific implementations, for the push-down execution of target queries on large-scale data, a distributed execution approach can be adopted. This fully utilizes the parallel processing capabilities of the storage location, decomposing the execution task of the target query into multiple subtasks, allowing each subtask to execute in parallel, thereby improving the execution efficiency of the target query. Furthermore, a reasonable data block size can be allocated to each subtask to ensure load balancing among them, avoiding both overload and idle periods.

[0145] Figure 8 The flowchart provided in this application illustrates the process of converting data operations in query plan information into target query statements, as follows: Figure 8 As shown, in step S702 above, the data operations in the query plan information are converted into the target query statement according to the target language type, including the following sub-steps:

[0146] S801. Identify the first operation and the second operation from the query plan information; wherein the first operation represents a data operation that allows pushdown, and the second operation represents a data operation that does not allow pushdown.

[0147] For example, the first and second operations are identified from the query plan information. The first operation represents a data operation that allows pushdown, while the second operation represents a data operation that does not allow pushdown. For instance, data read operations, simple conditional filtering operations, and simple data transformation operations do not involve cross-table joins (i.e., they require processing data across different tables), and therefore can be pushed down as the first operation. Complex aggregation operations and join operations that require extracting multiple data blocks involve cross-table joins; therefore, these operations cannot be pushed down and are classified as the second operation.

[0148] For example, if the query plan only contains keywords such as SELECT, FROM, and WHERE from the SQL-based query language, it consists of simple selection and filtering operations. It only requires reading data from specific columns in the data table and performing simple filtering on the data in the data table. Therefore, these operations are the first operations that can be pushed down.

[0149] If the query plan contains keywords such as JOIN, UNION, and GROUP BY in the SQL-based query language, then the data operations involve joins between multiple data tables. Therefore, these operations are non-pushdownable second operations.

[0150] Furthermore, the types of operations supported by the data source's storage location can be considered, and only data operations supported by the data source's storage location can be pushed down. For example, for data stored in a distributed file system, when performing calculation pushdown, it is necessary to consider whether the storage location in the distributed file system supports filtering and transformation operations.

[0151] S802. Based on the target language type, convert the first operation in the query plan information into the target query statement.

[0152] For example, based on the target language type, the first operation in the query plan information is converted into the target query statement. That is, each first operation in the query plan information is converted into a corresponding target query statement. The target query statement can then be pushed down to the data source for execution.

[0153] S803: Retain the second operation in the query plan information into the preset query engine.

[0154] For example, the second operation in the query plan information is retained in a preset engine. That is, each second operation does not perform a pushdown operation, but is executed in the preset engine of the data operation calling end.

[0155] For example, data operations that connect historical data tables in a distributed file system with customer information and sales order tables in a relational database are stored in a pre-defined SQL engine. Data operations that read data from the distributed file system and filter data based on the range of field values ​​are then pushed down to the storage area in the distributed file system. It should be noted that since the query engine in the distributed file system is different from the pre-defined SQL engine of the upper-level calling end, the data operations pushed down to the storage area in the distributed file system need to be converted into target query statements supported by the query engine in the distributed file system for execution. After the target query statement is executed, the data processing results are returned in DataFrame format, facilitating in-depth analysis of production costs, sales profits, and customer purchasing behavior for different product lines by the calling end. This provides strong support for enterprises to optimize production resource allocation, formulate marketing strategies, and improve the accuracy of financial decisions.

[0156] In this embodiment, a first operation and a second operation are identified from the query plan information. Based on the target language type, the first operation in the query plan information is converted into the target query statement, and the second operation in the query plan information is retained in a preset engine. This prevents data operations that cannot be executed at the data source from being forcibly pushed down, thus avoiding operation failures or errors. It also ensures that all data operations that can be pushed down are pushed down, resulting in performance improvements. With pushdown operations, not only are data transmission and intermediate result storage costs reduced, but also upper-layer advanced analytics applications can fully utilize the optimized computing power of the lakeware integrated architecture engine, improving data access efficiency.

[0157] In some specific implementations, after a data operation is successfully pushed down, each data operation is executed in its corresponding engine and the data processing result is returned. It should be noted that the default data format of the returned data processing result is a DataFrame, but the database client can also convert the returned data processing result to other data formats, such as JSON or array data formats. Furthermore, if the data processing results returned by multiple data operations meet preset combination conditions, the client can also perform further processing operations such as joining and aggregating the data processing results returned by multiple data operations.

[0158] Figure 9 This is a schematic diagram of the structure of the data access device based on multiple data sources provided in this application, such as... Figure 9 As shown, the data access device 90 based on multiple data sources provided in this embodiment includes:

[0159] The receiving module 901 is used to receive data attribute information sent by the user;

[0160] The first generation module 902 is used to generate declarative interface information based on data attribute information; wherein, the declarative interface information is a declarative data processing interface that represents the data that the user wants to access.

[0161] The second generation module 903 is used to generate query plan information based on the declarative interface information; wherein, the query plan information includes the data operations to be performed when accessing the data to be accessed;

[0162] The pushdown module 904 is used to push down the data operations in the query plan information to the data source of the data to be accessed, so as to obtain the data to be accessed.

[0163] In one possible implementation, the data attribute information includes the data source type of the data to be accessed; the first generation module 902 is further configured to:

[0164] Based on the data source type, determine the data source connection method corresponding to the data source type; where the data source connection method represents the way to connect to the database in the data source.

[0165] Based on the data attribute information and the data source connection method corresponding to the data source type, declarative interface information is generated based on preset operation functions.

[0166] In one possible implementation, the first generation module 902 is further configured to:

[0167] Based on data attribute information and data source connection methods corresponding to data source types, task flow orchestration information is generated based on preset operation functions; among which, task flow orchestration information represents the dependencies and execution order between data operations;

[0168] Based on the task flow orchestration information, generate declarative interface information.

[0169] In one possible implementation, the second generation module 903 is further configured to:

[0170] The declarative interface information is parsed to construct an abstract syntax tree; the abstract syntax tree represents the declarative interface information, and the nodes in the abstract syntax tree represent data operations.

[0171] Generate query plan information based on the abstract syntax tree.

[0172] In one possible implementation, the second generation module 903 is further configured to:

[0173] Lexical analysis is performed on the declarative interface information, decomposing the declarative interface information into at least one lexical unit;

[0174] Syntactic analysis is performed on each lexical unit to construct an abstract syntax tree.

[0175] In one possible implementation, the second generation module 903 is further configured to:

[0176] The nodes in the abstract syntax tree are partitioned to obtain at least one combination of nodes;

[0177] For each node combination, a query statement fragment is generated; where the query statement fragment represents the data operation in the node combination;

[0178] Based on the order of nodes in the abstract syntax tree, the query statement fragments are combined into query plan information.

[0179] In one possible implementation, the pushdown module 904 is also used for:

[0180] Determine the storage medium where the data source of the data to be accessed is located, and determine the target language type corresponding to the storage medium; whereby the target language type represents the type of query language supported by the storage medium;

[0181] Based on the target language type, the data operations in the query plan information are converted into the target query statement;

[0182] Based on the preset query engine, the target query statement is pushed down to the data source of the data to be accessed, and the data to be accessed is obtained.

[0183] In one possible implementation, the pushdown module 904 is also used for:

[0184] Identify the first operation and the second operation from the query plan information; wherein, the first operation represents a data operation that allows pushdown, and the second operation represents a data operation that does not allow pushdown;

[0185] Based on the target language type, the first operation in the query plan information is converted into the target query statement;

[0186] The second operation in the query plan information is retained in the preset query engine.

[0187] The data access device based on multiple data sources provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0188] Figure 10 This is a schematic diagram of the structure of the data access device based on multiple data sources provided in this application. Figure 10 As shown, the data access device 100 based on multiple data sources provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the data access device 100 based on multiple data sources further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus.

[0189] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0190] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0191] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0192] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0193] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0194] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0195] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0196] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0197] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0198] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0199] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0201] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0203] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data access method based on multiple data sources, characterized in that, include: Receive data attribute information sent by the user; Based on the data attribute information, declarative interface information is generated; wherein, the declarative interface information is a declarative data processing interface that represents the data that the user wants to access; Based on the declarative interface information, query plan information is generated; wherein, the query plan information includes the data operations to be performed when accessing the data to be accessed; The data operations in the query plan information are pushed down to the data source of the data to be accessed, thus obtaining the data to be accessed.

2. The method according to claim 1, characterized in that, The data attribute information includes the data source type of the data to be accessed; the step of generating declarative interface information based on the data attribute information includes: Based on the data source type, determine the data source connection method corresponding to the data source type; wherein, the data source connection method represents the way to connect to the database in the data source; Based on the data attribute information and the data source connection method corresponding to the data source type, the declarative interface information is generated based on a preset operation function.

3. The method according to claim 2, characterized in that, The step of generating the declarative interface information based on the data attribute information and the data source connection method corresponding to the data source type, and based on a preset operation function, includes: Based on the data attribute information and the data source connection method corresponding to the data source type, task flow orchestration information is generated based on preset operation functions; wherein, the task flow orchestration information represents the dependencies and execution order between data operations; Based on the task flow orchestration information, the declarative interface information is generated.

4. The method according to claim 1, characterized in that, The step of generating query plan information based on the declarative interface information includes: The declarative interface information is parsed to construct an abstract syntax tree; wherein, the abstract syntax tree represents the declarative interface information, and the nodes in the abstract syntax tree represent data operations; The query plan information is generated based on the abstract syntax tree.

5. The method according to claim 4, characterized in that, The step of parsing the declarative interface information and constructing an abstract syntax tree includes: Lexical analysis is performed on the declarative interface information to decompose it into at least one lexical unit. Syntactic analysis is performed on each lexical unit to construct the abstract syntax tree.

6. The method according to claim 4, characterized in that, The step of generating the query plan information based on the abstract syntax tree includes: The nodes in the abstract syntax tree are divided to obtain at least one combination of nodes; For each node combination, a query statement fragment is generated; wherein, the query statement fragment represents the data operation in the node combination; Based on the order of nodes in the abstract syntax tree, the query statement fragments are combined into the query plan information.

7. The method according to claim 1, characterized in that, The step of pushing down the data operation in the query plan information to the data source of the data to be accessed, to obtain the data to be accessed, includes: Determine the storage medium where the data source of the data to be accessed is located, and determine the target language type corresponding to the storage medium; wherein, the target language type represents the type of query language supported by the storage medium; Based on the target language type, the data operations in the query plan information are converted into the target query statement; Based on a preset query engine, the target query statement is pushed down to the data source of the data to be accessed, thereby obtaining the data to be accessed.

8. The method according to claim 7, characterized in that, The step of converting the data operations in the query plan information into a target query statement according to the target language type includes: Identify a first operation and a second operation from the query plan information; wherein the first operation represents a data operation that allows pushdown, and the second operation represents a data operation that does not allow pushdown; Based on the target language type, the first operation in the query plan information is converted into the target query statement; The second operation in the query plan information is retained in the preset query engine.

9. A data access device based on multiple data sources, characterized in that, include: The receiving module is used to receive data attribute information sent by the user; The first generation module is used to generate declarative interface information based on the data attribute information; wherein, the declarative interface information is a declarative data processing interface that represents the data that the user wants to access; The second generation module is used to generate query plan information based on the declarative interface information; wherein, the query plan information includes the data operations to be performed when accessing the data to be accessed; The push-down module is used to push down the data operations in the query plan information to the data source of the data to be accessed, so as to obtain the data to be accessed.

10. A data access device based on multiple data sources, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.