Data processing method and system, electronic device and readable storage medium

Abstract

The application discloses a data processing method, system, electronic equipment and readable storage medium, and belongs to the technical field of data processing. The method comprises the following steps: converting an obtained query language into a data query request representation according to at least one query clause; generating a query plan representation according to the data query request representation and at least one execution operator; and obtaining a data processing result by executing a data processing operation corresponding to the query plan representation, wherein the data processing operation comprises a data writing operation and a data query operation. In this way, the query language and the execution logic can be decoupled, so that the same type of query language can be executed using different execution logics, or the same execution logic can process different types of query languages, thereby improving the reusability of query execution.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, system, electronic device and readable storage medium. Background Art

[0002] Existing data management systems typically include database systems such as MySQL and MongoDB, as well as big data management systems such as Snowflake and Spark. These systems typically use their own SQL dialects or proprietary APIs to add, delete, modify, and query data. These data management systems typically include a query engine, an execution engine, and a storage engine. The query engine translates the SQL dialect into an abstract syntax tree and generates optimized query plans. The execution engine executes the query plans, and the storage engine handles data access and transaction management. These tightly coupled modules form the respective data management systems.

[0003] Although most data management systems have logically similar components, existing database management systems are developed and maintained as a whole. The specific implementations of these modules are highly decentralized and have almost no reusability, which causes existing data management systems to still have problems such as fragmentation, repeated development, and high maintenance costs. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, system, electronic device, and readable storage medium to at least solve the problem of low component reusability in related data management systems.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a data processing method, comprising: converting an acquired query language into a data query request representation according to at least one query clause; generating a query plan representation according to the data query request representation and at least one execution operator; and obtaining a data processing result by executing a data processing operation corresponding to the query plan representation, wherein the data processing operation includes a data write operation and a data query operation.

[0007] In a second aspect, an embodiment of the present application provides a data processing system, comprising: an interface intermediate representation layer module, for converting the acquired query language into a data query request representation based on at least one query clause; a plan intermediate representation layer module, for generating a query plan representation based on the data query request representation and at least one execution operator; and a result acquisition module, for obtaining data processing results by executing data processing operations corresponding to the query plan representation, wherein the data processing operations include data writing operations and data query operations.

[0008] In a third aspect, an electronic device is provided, which includes a processor and a memory. The memory stores programs or instructions executable on the processor. When the programs or instructions are executed by the processor, the steps of the method according to the first aspect are implemented.

[0009] In a fourth aspect, a computer readable storage medium is provided, which stores programs or instructions. When the programs or instructions are executed by a processor, the steps of the method according to the first aspect are implemented.

[0010] In a fifth aspect, a computer program product is provided, which includes a computer program stored on a non-transitory computer readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform the steps of the method according to the first aspect.

[0011] In the embodiments of the present application, the obtained query language is converted into a data query request representation according to at least one query clause; a query plan representation is generated according to the data query request representation and at least one execution operator; and a data processing result is obtained by performing a data processing operation corresponding to the query plan representation, where the data processing operation includes a data write operation and a data query operation. In this way, by converting the query language into the data query request representation and generating the query plan representation according to the data query request representation, the query language can be decoupled from the execution logic, so that the same type of query language can be executed using different execution logics, or the same execution logic can process different types of query language, thereby improving the reusability of query execution.

[0012] It should be understood that the general description above and the detailed description below are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 An architecture diagram of a related data management system is shown;

[0015] Figure 2 A flowchart of a data processing method provided by some embodiments of the present application is shown;

[0016] Figure 3 An architecture diagram of a data management system provided by some embodiments of the present application is shown;

[0017] Figure 4 A schematic diagram showing the structure of a data processing system provided by some embodiments of the present application is shown;

[0018] Figure 5 A schematic diagram of the architecture of an assembled data management system provided by some embodiments of the present application is shown;

[0019] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0021] like Figure 1 As shown, the relevant data management system usually includes an interface layer 110, a query engine 120, an execution engine 130 and a storage engine 140; wherein, the interface layer 110 is used to obtain the query language input by the user; the query engine 120 is used to translate the query language into an abstract syntax tree and generate a query plan; the execution engine 130 is used for the specific execution of the query plan; the storage engine 140 is used for data access and transaction management during the execution process. These modules are tightly coupled together to form a data management system. Although most data management systems have similar components logically, existing database management systems are developed and maintained as a whole. Various database management systems usually use their own query languages ​​or private interfaces to perform data addition, deletion, modification and query operations. For example, relational databases use the standard SQL query language for data operations; NoSQL databases use their own private query languages ​​or command interfaces to flexibly adapt to various data models; and graph databases use graph query languages ​​to operate on graph data. The query language or interface of each database management system is optimized for its data model and architecture, so their addition, deletion, modification and query operations are usually not interoperable. This fragmentation situation has at least the following problems:

[0022] 1) Developers need to reinvent the wheel between different systems, increasing development and maintenance costs;

[0023] 2) The lack of unified standards across different data management systems forces users to learn and adapt to multiple incompatible Structured Query Language (SQL) and non-SQL dialects, increasing their cognitive burden and learning costs.

[0024] 3) The functions and semantics of different systems are inconsistent. Users need to constantly adapt to different behaviors and features when using multiple systems, which reduces work efficiency.

[0025] Because each system has its own unique Application Programming Interface (API) and features, users need to spend a lot of time learning and mastering how to use different systems. At the same time, due to the lack of reusability between the various modules of different systems, the development of new systems needs to start from scratch, resulting in long development cycles and difficulty in quickly launching new features and improvements. In order to quickly launch prototypes, developers often sacrifice code stability and maintainability, resulting in the continuous accumulation of technical debt. Moreover, due to the high fragmentation of data management systems, hardware vendors find it difficult to optimize for specific data processing needs, resulting in low collaborative efficiency between hardware and software.

[0026] In response to the problems existing in the above-mentioned data processing process, an embodiment of the present application provides a data processing method. This method decouples the query language from the execution logic by converting the query language into a data query request representation and generating a query plan representation based on the data query request representation, so that the same type of query language can be executed using different execution logic, or the same execution logic can process different types of query languages, thereby improving the reusability of query execution.

[0027] See also Figure 2 , Figure 2 A flow chart of the data processing method provided in some embodiments of the present application is shown. The execution subject of the method can be a terminal device or a server, wherein the terminal device can be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, and the terminal device can be a terminal device used by a user. The server can be an independent server, or a server cluster composed of multiple servers, and the server can be a background server of a certain business, or a background server of a certain platform (for example, a data management system, a data processing platform, a query system, etc.). In the embodiment of the present application, the execution subject is taken as an example to illustrate. For the case of the terminal device, it can be processed according to the following relevant content, which will not be repeated here. As shown in the figure, the data processing method 200 may include the following steps:

[0028] Step 201: Convert the acquired query language into a data query request representation according to at least one query clause.

[0029] The query language in step 201 includes Structured Query Language (SQL), Graph Query Language (GraphQL), and natural language; the query clauses in step 201 include the clause select_fields (SELECT statement) for indicating the fields returned by the query, the clause from_tables (FROM statement) for indicating the table of data source, the clause join_conditions (JOIN statement) for indicating the table connection condition, the clause where_conditions (WHERE statement) for indicating the filter condition of the query, the clause group_by (GROUP BY statement) for indicating the grouping field, the clause having_conditions (HAVING statement) for indicating the filter field after grouping, and the clause order_by (ORDER BY statement) for indicating the sorting rule. BY statement), the limit clause (LIMIT statement) used to indicate the limit on the number of rows returned by the query, the offset clause (OFFSET statement) used to indicate the query offset, the aggregations clause used to indicate aggregation operations (such as SUM, COUNT, AVG), the distinct clause used to indicate whether to remove duplicates, and the subqueries clause used to indicate subqueries.

[0030] In specific implementation, Figure 3 As shown, the interface layer 310 of the data management system can include various forms of external interfaces, including SQL, GraphQL, Pandas, REST, natural language interfaces, etc., and various types of query languages ​​are obtained through the interface layer 310. For example, if the natural language input by the user is "Find 10 books on natural language processing, sorted by publication date in descending order", if the natural language interface is used for query, the natural language can be converted into an interface intermediate representation (IR) using a trained model. The converted prompt word (prompt) format is {convert "Find 10 books on natural language processing, sorted by publication date in descending order" into the interface IR format}; if the SQL interface is used for query, the SQL statement is as follows:

[0031] SQL statement:

[0032] SELECT title, author, publish_date

[0033] FROM books

[0034] WHERE topic = 'Natural Language Processing'

[0035] ORDER BY publish_date DESC

[0036] LIMIT 10;

[0037] According to at least one preset query clause, such as select_fields, from_tables, where_conditions, etc., the obtained natural language, SQL and other query languages ​​are converted into a more abstract intermediate representation. The intermediate representation mainly contains basic query information, such as select fields, where conditions, orderBy sorting and limit restrictions.

[0038] In this way, the query language input by the user can be converted into a data query request representation that can be understood by a machine through at least one query clause, so as to facilitate the subsequent execution of the data processing operation corresponding to the query language.

[0039] Step 202: Generate a query plan representation according to the data query request representation and at least one execution operator.

[0040] Among them, the execution operators in the above step 202 include the operator Scan for indicating a scan operation, the operator Filter for indicating a filter operation, the operator Projection for indicating a projection operation, the operator Join for indicating a join operation, the operator Aggregation for indicating an aggregation operation, the operator Sort for indicating a sorting operation, the operator Limit for indicating a restriction operation, and the operator Union for indicating a union operation.

[0041] In a specific implementation, a query plan representation is generated based on the data query request representation converted in the above step 201 and execution operators such as Scan, Filter, and Projection. The query plan representation contains specific operators, such as Scan, Sort, and Limit, which describe the execution order of the query.

[0042] In this way, by further converting the data query request representation into a query plan representation, the assembled hierarchical design in the query plan representation can ensure the flexibility and scalability of the query while ensuring the maximization of execution efficiency.

[0043] Step 203: Obtain data processing results by executing data processing operations corresponding to the query plan representation, wherein the data processing operations include data writing operations and data query operations.

[0044] In a specific implementation, the execution engine can be called to execute the data processing operation corresponding to the query plan, and the data processing result output by the execution engine can be obtained. Figure 3 As shown, the execution engine 330 may include Velox, spark, Ray, Postgre, Flink, etc. Here, the application scenarios of different execution engines are different. For example, Velox focuses on query engines and is suitable for large-scale data analysis; Spark has a powerful big data analysis framework and is suitable for batch processing and stream processing; Ray focuses on distributed computing, especially machine learning and deep learning tasks; PostgreSQL is a powerful relational database suitable for complex database applications; Flink focuses on stream processing and is suitable for real-time data computing and event-driven applications. Since the embodiment of the present application converts the query language input by the user into a unified query plan representation through query clauses and execution operators, the unified query plan representation can be adapted to multiple execution engines.

[0045] Through the above steps, the query language is converted into a data query request representation, and a query plan representation is generated based on the data query request representation. This can decouple the query language from the execution logic, so that the same type of query language can be executed using different execution logics, or the same execution logic can process different types of query languages, thereby improving the reusability of query execution.

[0046] In some embodiments, in step 201 above, converting the acquired query language into a data query request representation according to at least one query clause includes:

[0047] Obtaining a query language; generating a structured representation corresponding to the query language by parsing the query language; mapping the structured representation to at least one query clause to generate a data query request representation.

[0048] In a specific implementation, the query language input by the user is obtained through the interface layer 310 of the data management system, and the interface layer 310 includes interfaces such as SQL, GraphQL, and natural language. The query language is parsed and structured, for example, the query language is represented as an expression tree, including function calls, table references, constants, and various operator operations, such as filtering, projection, sorting, joining, aggregation, window functions, shuffling / repartitioning, etc.; then, the structured representation is mapped to query clauses such as select_fields, from_tables, where_conditions, etc. to generate a data query request representation, which includes basic query information, such as select fields, where conditions, orderBy sorting, and limit restrictions.

[0049] In some possible implementations, the above-mentioned generation of a structured representation corresponding to the query language by parsing the query language includes:

[0050] Obtain multiple clauses in the query language; convert the query elements in each clause into an interface intermediate representation; reorganize the interface intermediate representation based on the dependency relationship between the multiple clauses to generate a structured representation corresponding to the query language.

[0051] In specific implementations, the query language includes multiple clauses, such as SELECT, FROM, WHERE, JOIN, etc. Multiple clauses in the query language are obtained, and query elements such as fields, aggregate functions, and expressions in each clause are converted into an interface intermediate representation (IR). Based on the dependencies between multiple clauses, the interface IR is reorganized to generate a structured representation corresponding to the query language. The mapping method from query language to interface IR is shown in the following table:

[0052] Table 1. Mapping methods from query language to interface IR

[0053]

[0054] In some embodiments, in step 202 above, generating a query plan representation based on the data query request representation and at least one execution operator includes:

[0055] A query logical plan is generated based on a target query clause in a data query request representation; wherein the target query clause includes a first query clause for indicating query intent, a second query clause for indicating contextual information, and a third query clause for indicating a query structure; an executable physical plan is generated by optimizing the execution efficiency of the query logical plan; and the executable physical plan is converted into a query plan representation based on at least one execution operator.

[0056] In specific implementation, Figure 3As shown, the query engine 320 includes Calcite, Orca, Presto, Postgre, Flink, etc. The query engine receives the data query request representation in the above step 201, and first extracts the first query clause for indicating the query intent, the second query clause for indicating the context information, and the third query clause for indicating the query structure from the data query request representation, such as select_fields, join_conditions, where_conditions, order_by, etc., to generate a query logical plan; based on time cost and preset rules, the execution efficiency of the query logical plan is optimized to generate an executable physical plan; then, according to the execution operators such as Scan, Sort, Limit, etc., the executable physical plan is converted into a query plan representation, that is, a query plan IR. The query plan IR describes the execution method and optimization strategy of the query. The plan IR consists of a series of operators (Operators), which represent the execution steps of the database query. The plan IR is defined using json and includes the following content:

[0057] root: the root operator of the query (final output)

[0058] operators: the various operators involved in query execution

[0059] type: operator type (such as Scan, Filter, Sort, Limit, etc.)

[0060] Input: The upstream operator that the operator depends on (i.e., the data source)

[0061] output_fields: the fields output by the operator

[0062] conditions: filtering, connection and other conditions

[0063] order_by: sorting method

[0064] limit: The number of rows returned by the query

[0065] cost: Estimated execution cost of the operator (optional)

[0066] parallelism: the degree of parallelism of the operator (optional)

[0067] Among them, operator types mainly include:

[0068]

[0069] In some embodiments, in step 203, obtaining a data processing result by executing a data processing operation corresponding to the query plan representation includes:

[0070] Obtain at least one operator node in the query plan representation and assign an execution task to each of the operator nodes; select a target storage engine that matches the execution task from a plurality of preset storage engines, and adapt the execution task to an operation request of the target storage engine; and obtain a data processing result by calling the target storage engine to execute a data processing operation corresponding to the operation request.

[0071] In specific implementation, Figure 3 As shown, the execution engine 330 can have different options according to different scenarios, including Velox, spark, Ray, Postgre, Flink, etc. The execution engine 330 is the core component in the data management system that is responsible for actually executing the query plan and returning the results. It first parses the operator nodes in the query plan IR and assigns execution tasks to each operator. Optionally, the execution engine 330 can manage the execution order and dependencies of tasks through a task scheduler, divide the tasks into multiple subtasks and assign them to different computing nodes or threads for parallel processing; each task executes operators such as scanning, filtering, joining, sorting, aggregation, and restriction in turn, and generates intermediate results by reading data from the storage engine and gradually converting and processing it; the execution engine 330 is used to manage memory and intermediate results. For large-scale data processing, the intermediate results may be written to the disk cache to avoid memory overflow.

[0072] In specific applications, in order to improve efficiency, the above-mentioned execution engine 330 can use vectorized execution technology to process batch data operations as units, and can combine JIT compilation, partitioned parallelism, pipeline optimization and other strategies to optimize performance. During the execution process, the execution engine 330 can also detect and handle runtime errors and provide a fault-tolerant mechanism for automatic retry or degradation processing. Finally, the execution engine 330 merges and formats the intermediate results of all subtasks to obtain the data processing results, and returns the data processing results to the user or upper-level application in the output format specified by the query plan IR.

[0073] This approach ensures efficient and stable query execution while offering excellent scalability and flexibility. By extending the execution engine, it supports partially or fully offloading execution tasks (e.g., projection, aggregation, sorting, encoding and decoding) to heterogeneous hardware such as GPUs, FPGAs, and DPUs, facilitating support for new heterogeneous hardware.

[0074] Among them, such as Figure 3As shown, the storage engine 340 can include different components such as DuckDB, RocksDB, SQLite, LanceDB, Parquet, etc. according to different execution scenarios, which are the core components responsible for the persistent storage and efficient access of data in the data management system. The storage engine 340 supports various storage structures such as row storage, column storage and key-value storage, and provides corresponding optimization strategies according to the characteristics of different storage engines. Further, a storage adaptation layer is added between the execution engine 330 and the storage engine 340, which is used to select a target storage engine matching the execution task from a plurality of preset storage engines, and adapt the execution task to an operation request of the target storage engine. The storage engine 340 executes a data processing operation corresponding to the operation request by calling the target storage engine to obtain a data processing result. Optionally, the storage engine 340 can also improve data access performance through index structure, batch operation and data compression technology, and ensure data persistence and fault recovery capability, for example, through Write-Ahead Logging (WAL), transaction log, snapshot and checkpoint mechanism to realize data reliability. In order to support multi-user concurrent access, the storage engine 340 can also provide a lock mechanism and Multi-Version Concurrency Control (MVCC) to ensure the isolation and consistency of transactions. In this way, not only efficient data storage and management capabilities are provided, but also flexible storage engine replacement and extension are supported, ensuring high performance and scalability of the system in different application scenarios.

[0075] In some possible implementation ways, the above selecting a target storage engine matching the execution task from a plurality of preset storage engines includes:

[0076] According to the characteristic information of the plurality of storage engines, determining a task scenario matching each storage engine; and selecting a target storage engine matching a scenario where the execution task is located from the plurality of storage engines according to a matching relationship between the plurality of storage engines and the task scenario.

[0077] In practice, the storage adaptation layer serves as an abstraction layer between the execution engine and the underlying storage engine. Its primary function is to provide the execution engine with a unified data access interface, enabling seamless integration with different storage engines (e.g., DuckDB, SQLite, RocksDB, LanceDB, etc.). The storage adaptation layer's core responsibilities include storage engine abstraction, data format conversion, interface standardization, optimization strategy adaptation, and resource management. First, the storage adaptation layer defines standardized storage engine interfaces, such as CreateTable, DropTable, Scan, IndexScan, Read, Write, Update, and Delete, and provides specific adapter implementations for each storage engine. Second, it is responsible for adapting execution engine requests to the underlying storage engine's data format and API, including data encoding and decoding, metadata management, and table structure mapping. The storage adaptation layer also provides optimization strategy adaptation, allowing the execution engine to leverage storage engine characteristics to determine the most suitable workload for each storage engine. For example, RocksDB is suitable for key-value indexing, DuckDB for column storage, and SQLite for transaction support. Furthermore, based on the matching relationship between multiple storage engines and task scenarios, a target storage engine that matches the scenario where the task is executed can be selected from multiple storage engines to improve query performance.

[0078] Optionally, to support concurrent execution and efficient data access, the storage adaptation layer can also provide caching mechanisms, parallel I / O scheduling, memory management, and handle exceptions and errors that may be thrown by the underlying storage engine. This allows the execution engine to uniformly access and control different storage engines, achieving efficient and flexible storage engine adaptation and integration.

[0079] By combining different components, we can meet the data management needs of different scenarios. For example, in transaction processing scenarios, we use SQL as the interface, PostgreSQL as the query and execution engine, and SQLite as the storage engine. For data analysis scenarios, we use GraphQL as the interface, Orca or Velox as the query and execution engines, and DuckDB as the storage engine. In large model application scenarios, we use natural language and Pandas as the interface, Calcite or Ray as the query and execution engines, and Parquet as the storage engine, as shown in the following table:

[0080] Table 2. Combination relationship table of different components in the data management system

[0081]

[0082] In this way, through the combination of different components, the data management software can be expanded to meet the needs of different scenarios.

[0083] Figure 4 The data processing system provided by some embodiments of the present application is shown in FIG. Figure 2 In the embodiment shown, all or part of the data processing system 400 includes:

[0084] The interface intermediate representation layer module 410 is configured to convert the acquired query language into a data query request representation according to at least one query clause;

[0085] A plan intermediate representation layer module 420 is configured to generate a query plan representation based on the data query request representation and at least one execution operator;

[0086] The result acquisition module 430 is used to acquire data processing results by executing data processing operations corresponding to the query plan representation, wherein the data processing operations include data writing operations and data query operations.

[0087] In some embodiments, the interface intermediate representation layer module 410, when used to convert the acquired query language into a data query request representation according to at least one query clause, is specifically used to:

[0088] Get query language;

[0089] By parsing the query language, generating a structured representation corresponding to the query language;

[0090] The structured representation is mapped to at least one query clause to generate a data query request representation.

[0091] In some possible implementations, the interface intermediate representation layer module 410, when used to parse the query language and generate a structured representation corresponding to the query language, is specifically configured to:

[0092] Obtaining multiple clauses in the query language;

[0093] Convert query elements in each clause into an interface intermediate representation;

[0094] The interface intermediate representation is reorganized according to the dependency relationship between the multiple clauses to generate a structured representation corresponding to the query language.

[0095] In some embodiments, the plan intermediate representation layer module 420, when used to generate a query plan representation according to the data query request representation and at least one execution operator, is specifically used to:

[0096] Generating a query plan representation according to the data query request representation and at least one execution operator includes:

[0097] Generate a query logic plan based on a target query clause in the data query request; wherein the target query clause includes a first query clause for indicating query intent, a second query clause for indicating context information, and a third query clause for indicating query structure;

[0098] Generate an executable physical plan by optimizing the execution efficiency of the query logical plan;

[0099] The executable physical plan is converted into a query plan representation according to at least one execution operator.

[0100] In some embodiments, the result acquisition module 430 includes:

[0101] An execution engine layer module, configured to obtain at least one operator node in the query plan representation and assign an execution task to each operator node;

[0102] A storage adaptation layer module is configured to select a target storage engine that matches the execution task from a plurality of preset storage engines, and adapt the execution task to an operation request of the target storage engine;

[0103] The storage engine layer module is used to execute the data processing operation corresponding to the operation request by calling the target storage engine to obtain the data processing result.

[0104] In an exemplary embodiment, Figure 5 As shown, the embodiment of the present application further provides an assembled data management system, the assembled data management system comprising:

[0105] The interface layer 510 is used to obtain the query language input by the user;

[0106] An interface intermediate representation 520, configured to convert the acquired query language into a data query request representation according to at least one query clause;

[0107] A query engine 530, configured to generate a query plan representation according to the data query request representation and at least one execution operator;

[0108] Plan intermediate representation 540, used to obtain the query plan representation generated by the query engine 530;

[0109] An execution engine 550 is configured to obtain at least one operator node in the query plan and assign an execution task to each operator node;

[0110] The storage adaptation layer 560 is configured to select a target storage engine that matches the execution task from a plurality of preset storage engines, and adapt the execution task to an operation request of the target storage engine;

[0111] The storage engine layer 570 is configured to execute a data processing operation corresponding to the operation request by calling the target storage engine to obtain a data processing result.

[0112] The embodiment of the present application decomposes the data management system into a series of reusable components, including an interface layer, an interface intermediate representation (IR), a query engine, a plan intermediate representation (IR), an execution engine, a storage adapter, and a storage engine. These components interact through clearly defined interfaces (APIs), thereby achieving modularity and decoupling of the system. This modular design not only improves development efficiency, but also reduces the maintenance cost of the system, while providing users with a more consistent experience. Through a unified interface IR and plan IR, different system interfaces can generate a unified interface IR, which is then used by the query engine to generate a unified optimized plan IR, and different execution engines can execute these plan IRs. This decoupling allows the system interface, query engine, execution engine, and storage engine to develop independently, while also supporting cross-system query optimization and execution. This architecture not only supports a variety of workloads, from online transaction processing to online analytical processing, from stream processing to machine learning, but also allows developers to select and combine different components according to their needs, thereby quickly building a data management system that meets specific needs. Through componentization and standardization, data management systems will be able to better adapt to the rapidly changing technological environment, adapt to new hardware accelerators such as GPUs and FPGAs, promote the co-evolution between hardware and software, and give full play to the functional and performance advantages of new hardware.

[0113] The PostgreSQL database uses a plug-in mechanism (Extension) to extend the database functionality, allowing users to flexibly add new features and capabilities without modifying the core database code. Plug-ins can implement data type extensions, for example, by installing PostGIS to support geospatial data, or providing fuzzy search capabilities through pg_trgm. Plug-ins also support the definition of custom index methods, functions, and operators to optimize query performance. For example, the btree_gin plug-in enhances B-tree indexes. However, this plug-in mechanism usually relies on internal extensions and external function packages of the database. The extension capabilities are often tightly coupled to a specific database version and architecture, and cannot be shared and migrated across platforms between different database systems, bringing challenges in compatibility, maintenance, and performance. Compared to PostgreSQL's database extension method through a plug-in mechanism, the embodiment of the present application extends through an interface IR, a plan IR, and a storage adaptation layer. The interface IR and plan IR structurally separate query semantics from the execution plan, allowing the extension logic to be shared between different database systems. It has better flexibility, maintainability, and compatibility with heterogeneous systems, and is more suitable for building a modular, multimodal, and continuously evolving data processing platform.

[0114] Figure 6 A schematic diagram of the hardware structure of an electronic device that implements the embodiments of the present application is shown. Referring to this figure, at the hardware level, electronic device 600 includes a processor 610, and optionally, an internal bus 620, a network interface 630, and a memory. The memory may include internal memory 641, such as high-speed random-access memory (RAM), and may also include non-volatile memory 642, such as at least one disk storage device. Of course, electronic device 600 may also include hardware required for other services.

[0115] The processor 610, network interface 630, and memory can be interconnected via an internal bus 620. This internal bus 620 can be an Advanced Microcontroller Bus Architecture (AMDBA) bus, a Wishbone bus, an Open Core Protocol (OCP) bus, an Avalon bus, or the like. Such buses can be categorized as address buses, data buses, and control buses. For ease of illustration, this figure uses only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.

[0116] The memory stores programs. Specifically, the programs may include program codes, which include computer operating instructions. The memory may include internal memory 641 and non-volatile memory 642, and provides instructions and data to the processor 610.

[0117] The processor 610 reads the corresponding computer program from the non-volatile memory 642 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 610 executes the program stored in the memory and specifically performs the following: Figure 2 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0118] The above application Figure 2 The methods disclosed in the illustrated embodiments can be applied to or implemented by processor 610. Processor 610 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits or software instructions within processor 610. The processor 610 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly executed by a hardware decoding processor or by a combination of hardware and software modules within the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0119] The computer device can also execute the methods described in the above method embodiments and realize the functions and beneficial effects of the methods described in the above method embodiments, which will not be repeated here.

[0120] Of course, in addition to software implementation, the electronic device 600 of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, which, when executed by an electronic device including multiple application programs, enables the electronic device to execute Figure 2 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0122] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] Furthermore, an embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the following process is implemented: Figure 2 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0124] The embodiments of the present application can be applied to various electronic device collaboration or interconnection scenarios, including: collaboration and interconnection between mobile phones and laptops / tablets; collaboration and interconnection between mobile terminals and smart TVs / displays; collaboration and interconnection between mobile phones or tablets and in-car entertainment systems; collaboration and interconnection between mobile terminals and smart conference systems, etc., thereby meeting the diverse needs of users in scenarios such as smart homes, smart offices, and smart travel.

[0125] In short, the above description is only a preferred embodiment of the present application and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0126] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0127] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0129] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A data processing method, characterized in that: include: Converting the acquired query language into a data query request representation according to at least one query clause, wherein the query clause includes a clause for indicating fields returned by the query, a clause for indicating table join conditions, a clause for indicating filter conditions for the query, a clause for indicating grouping fields, a clause for indicating filter fields after grouping, a clause for indicating sorting rules, a clause for indicating a limit on the number of rows returned by the query, a clause for indicating a query offset, a clause for indicating an aggregation operation, a clause for indicating whether to perform deduplication, and a clause for indicating a subquery; Generate a query plan representation according to the data query request representation and at least one execution operator; Obtaining a data processing result by executing a data processing operation corresponding to the query plan representation, wherein the data processing operation includes a data writing operation and a data query operation; The step of converting the acquired query language into a data query request representation according to at least one query clause includes: acquiring a query language; generating a structured representation corresponding to the query language by parsing the query language; and mapping the structured representation to at least one query clause to generate a data query request representation. The generating of a structured representation corresponding to the query language by parsing the query language includes: obtaining multiple clauses in the query language; converting query elements in each clause into an interface intermediate representation; and reorganizing the interface intermediate representation according to dependency relationships between the multiple clauses to generate a structured representation corresponding to the query language; The method of obtaining a data processing result by executing a data processing operation corresponding to the query plan representation includes: obtaining at least one operator node in the query plan representation and assigning an execution task to each operator node; selecting a target storage engine that matches the execution task from a plurality of preset storage engines, and adapting the execution task to an operation request of the target storage engine; and obtaining a data processing result by calling the target storage engine to execute the data processing operation corresponding to the operation request.

2. The method according to claim 1, characterized in that Generating a query plan representation according to the data query request representation and at least one execution operator includes: Generate a query logic plan based on a target query clause in the data query request representation; wherein the target query clause includes a first query clause for indicating query intent, a second query clause for indicating context information, and a third query clause for indicating query structure; Generate an executable physical plan by optimizing the execution efficiency of the query logical plan; The executable physical plan is converted into a query plan representation according to at least one execution operator.

3. The method according to claim 1, characterized in that The selecting a target storage engine that matches the execution task from a plurality of preset storage engines includes: Determining a task scenario that matches each of the storage engines based on characteristic information of the preset multiple storage engines; According to the matching relationship between the plurality of storage engines and the task scenarios, a target storage engine matching the scenario where the task is to be executed is selected from the plurality of storage engines.

4. A data processing system, characterized in that: include: An interface intermediate representation layer module, configured to convert the acquired query language into a data query request representation based on at least one query clause, wherein the query clause includes a clause for indicating fields returned by the query, a clause for indicating table join conditions, a clause for indicating query filter conditions, a clause for indicating grouping fields, a clause for indicating filter fields after grouping, a clause for indicating sorting rules, a clause for indicating a limit on the number of rows returned by the query, a clause for indicating a query offset, a clause for indicating an aggregation operation, a clause for indicating whether to perform deduplication, and a clause for indicating a subquery; A plan intermediate representation layer module, configured to generate a query plan representation according to the data query request representation and at least one execution operator; a result acquisition module, configured to acquire data processing results by executing data processing operations corresponding to the query plan representation, wherein the data processing operations include data writing operations and data query operations; The interface intermediate representation layer module, when used to convert the acquired query language into a data query request representation according to at least one query clause, is specifically used to: Obtaining a query language; generating a structured representation corresponding to the query language by parsing the query language; mapping the structured representation to at least one query clause to generate a data query request representation; The interface intermediate representation layer module, when used to parse the query language and generate a structured representation corresponding to the query language, is specifically used to: obtain multiple clauses in the query language; convert query elements in each clause into an interface intermediate representation; and reorganize the interface intermediate representation based on the dependency relationships between the multiple clauses to generate a structured representation corresponding to the query language; The result acquisition module includes: An execution engine layer module, configured to obtain at least one operator node in the query plan representation and assign an execution task to each operator node; A storage adaptation layer module is configured to select a target storage engine that matches the execution task from a plurality of preset storage engines, and adapt the execution task to an operation request of the target storage engine; The storage engine layer module is used to execute the data processing operation corresponding to the operation request by calling the target storage engine to obtain the data processing result.

5. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images