Data processing method and device, storage medium, computer device, and program product

Abstract

The application provides a data processing method and device, a storage medium, a computer device and a program product. The method comprises: receiving a data operation statement for a data storage system; performing analysis processing on the data operation statement to obtain a first execution plan; converting a row scanning operator in the first execution plan into a vectorized scanning operator to obtain a second execution plan; starting the vectorized scanning operator in the second execution plan to read metadata columns in a metadata file corresponding to a data table to obtain metadata, and filling the metadata into a scanning filtering condition; filtering a business data file in the data storage system according to the scanning filtering condition filled with the metadata to obtain a target business data file corresponding to the data table, and performing columnar reading on target business data columns in the target business data file to obtain target business data. The application can realize vectorized scanning for a data table using columnar storage, and improve the efficiency and performance of data processing.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a data processing method, apparatus, storage medium, computer equipment, and program product. Background Technology

[0002] In today's digital age, the scale of big data is exploding, encompassing data from various fields such as the internet, finance, healthcare, and industry. This data is characterized by its massive volume, multiple sources, and heterogeneity, posing significant challenges to data storage and processing. To meet the demands of big data storage and analysis, various data storage and processing technologies have emerged.

[0003] Columnar storage has gradually become the mainstream method for big data storage due to its advantages in storage efficiency and query performance. However, current data computing engines typically use a row-based computing framework, which requires reorganizing column-stored data into row-stored data for computation. This column-to-row conversion process involves a large amount of data reorganization and memory operations, consuming a lot of computing resources and time, resulting in poor data processing efficiency and performance. Summary of the Invention

[0004] This application provides a data processing method, apparatus, storage medium, computer equipment, and program product that can perform vectorized scanning on data tables using columnar storage, thereby improving the efficiency and performance of data processing.

[0005] To solve the above-mentioned technical problems, this application provides the following technical solution: According to one aspect of this application, a data processing method is provided, comprising: Receive data manipulation statements for the data storage system; The data operation statement is parsed and processed to obtain a first execution plan. The first execution plan includes a row scan operator for a data table that uses columnar storage in the data storage system. The data table includes metadata columns and business data columns. The row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain a second execution plan. The scan output column of the vectorized scan operator includes the metadata column and the target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator. The vectorized scan operator in the second execution plan is started to read the metadata column in the metadata file corresponding to the data table to obtain the metadata, and the metadata is then filled into the scan filter conditions; The business data files in the data storage system are filtered according to the scanning filtering conditions filled with the metadata to obtain the target business data file corresponding to the data table, and the target business data columns in the target business data file are read in a columnar manner to obtain the target business data.

[0006] According to one aspect of this application, a data processing apparatus is provided, comprising: The receiving module is used to receive data operation statements for the data storage system; The parsing module is used to parse and process the data operation statement to obtain a first execution plan. The first execution plan includes a row scan operator for a data table that uses columnar storage in the data storage system. The data table includes a metadata column and a business data column. The optimization module is used to convert the row-based scan operator in the first execution plan into a vectorized scan operator to obtain a second execution plan. The scan output column of the vectorized scan operator includes the metadata column and the target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator. The execution module is used to start the vectorized scan operator in the second execution plan, read the metadata column in the metadata file corresponding to the data table to obtain metadata, and fill the metadata into the scan filter conditions; The execution module is further configured to filter the business data files in the data storage system according to the scanning filtering conditions filled with the metadata, obtain the target business data file corresponding to the data table, and perform columnar reading of the target business data columns in the target business data file to obtain the target business data.

[0007] In the above scheme, the metadata column includes a file path column, and the metadata includes the target file path read from the file path column in the metadata file. The execution module is further configured to: The business data files in the data storage system are filtered according to the scanning and filtering conditions filled with the target file path to obtain the business data file of the target file path; The target business data file corresponding to the data table is determined in the business data file of the target file path.

[0008] In the above scheme, the metadata column further includes a partition column, and the metadata also includes target partition information read from the partition column in the metadata file. The business data file of the target file path includes multiple partition business data files located in different partitions. The execution module is further configured to: The multiple partition service data files are filtered according to the scanning filtering conditions filled with the target partition information to obtain the partition service data file located in the target partition corresponding to the target partition information; The partition business data file located in the target partition is determined as the target business data file corresponding to the data table.

[0009] In the above scheme, the metadata column further includes a data location column, and the metadata also includes the target data location read from the data location column in the metadata file. The execution module is further configured to: The data records in the target business data file are filtered according to the scanning filtering conditions filled with the target data location to obtain the first target data record; The target business data is obtained by performing columnar reading on the target business data column in the first target data record.

[0010] In the above scheme, the optimization module is further used for: The first execution plan is subjected to complexity analysis to obtain the first complexity. The data table is subjected to complexity analysis to obtain the second complexity. Based on the first complexity and the second complexity, a prediction process is performed to obtain the predicted execution time of the first execution plan; When the predicted execution duration exceeds the duration threshold, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain the second execution plan.

[0011] In the above scheme, the number of data tables includes multiple tables, and the optimization module is further used for: Determine the proportion of operators corresponding to each of the multiple data tables in the first execution plan; Based on the first complexity and the second complexity corresponding to the multiple data tables, a prediction process is performed to obtain the execution time of the data tables corresponding to the multiple data tables. Using the proportion of operators corresponding to the multiple data tables as weights, the execution time of the data tables corresponding to the multiple data tables is weighted and summed to obtain the predicted execution time of the first execution plan.

[0012] In the above scheme, the metadata column and the target business data column are used to jointly construct the scanning filtering conditions of the vectorized scanning operator, and the execution module is further used for: Candidate business data is obtained by performing columnar reading on the target business data column in the target business data file, and the candidate business data is then filled into the scanning filter conditions. The data records in the target business data file are filtered according to the scanning and filtering conditions filled with the candidate business data to obtain the second target data record; The candidate business data in the second target data record is determined as the target business data.

[0013] In the above scheme, the target business data column includes a first target business data column and a second target business data column. The metadata column and the first target business data column are used to jointly construct the scanning filtering conditions of the vectorized scanning operator. The execution module is further used for: The first target business data column in the target business data file is read in a columnar manner to obtain candidate business data, and the candidate business data is then filled into the scanning filter conditions. The data records in the target business data file are filtered according to the scanning and filtering conditions filled with the candidate business data to obtain the third target data record; The candidate service data in the third target data record is determined as the first target service data; The second target business data column in the third target data record is read in a columnar manner to obtain the second target business data; The first target business data and the second target business data are identified as target business data.

[0014] In the above scheme, the number of vectorized scan operators includes multiple types, and the optimization module is further used for: The matching process is performed on multiple vectorized scan operators in the second execution plan; When the multiple vectorized scan operators in the second execution plan are successfully matched, the multiple vectorized scan operators in the second execution plan are merged into a target vectorized scan operator to obtain a third execution plan, and the target vectorized scan operator in the third execution plan is started; When the plurality of vectorized scan operators in the second execution plan fail to match, the plurality of vectorized scan operators in the second execution plan are started.

[0015] In the above scheme, the optimization module is further used for: When the data tables to be scanned by the multiple vectorized scan operators are the same, the scan output columns of the multiple vectorized scan operators have an inclusion relationship, and the scan filtering conditions of the multiple vectorized scan operators are the same, it is determined that the multiple vectorized scan operators have successfully matched.

[0016] In the above scheme, the optimization module is further used for: When the scan output columns of the plurality of vectorized scan operators are obtained by column clipping, the scan output column with the largest range is determined among the scan output columns of the plurality of vectorized scan operators, and the vectorized scan operator corresponding to the scan output column with the largest range is determined as the target vectorized scan operator; When the scan output columns of the plurality of vectorized scan operators are obtained without column clipping, any one of the vectorized scan operators is determined as the target vectorized scan operator.

[0017] In the above scheme, the multiple vectorized scan operators are located in different subqueries, and the subqueries also include computation operators other than the vectorized scan operators. The optimization module is further used for: Perform matching processing on the calculated operators included in multiple subqueries; When the computational operators included in the multiple subqueries are successfully matched, the vectorized scan operators included in the multiple subqueries are matched. The computational operators included in the multiple subqueries are merged into the target computational operator.

[0018] In the above scheme, the calculation operator includes a filtering operator, and the optimization module is further used for: When the filtering operators included in the multiple subqueries are at the same execution level, it is determined that the filtering operators included in the multiple subqueries are successfully matched. The target filtering operator is obtained by performing a union operation on the filtering operators included in the multiple subqueries.

[0019] In the above scheme, the computation operator includes a non-filtering operator, and the optimization module is further used for: When the non-filtering operators included in the multiple subqueries are at the same execution level, have the same operator type, and target the same data column, it is determined that the non-filtering operators included in the multiple subqueries are successfully matched. The non-filtered operator in any subquery is identified as the target non-filtered operator.

[0020] According to one aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the above-described data processing method.

[0021] According to one aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data processing method.

[0022] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions to implement the above-described data processing method.

[0023] This application receives data operation statements for a data storage system and triggers a three-stage processing phase: parsing, optimization, and execution. In the parsing phase, the data operation statements are parsed to form a first execution plan containing a series of computational operators. These operators include a row-based scan operator for a column-oriented data table in the data storage system. The data table includes metadata columns for storing metadata and business data columns for storing business data. The metadata provides descriptive information about the business data, used to locate it.

[0024] During the optimization phase, considering that columnar storage data tables can be read in columnar format, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain the second execution plan. The scan output columns of the vectorized scan operator include metadata columns and target business data columns. The metadata columns are used to construct the scan filtering conditions of the vectorized scan operator. Since the scan output columns of the vectorized scan operator cover the metadata columns, it is ensured that the scan filtering conditions constructed based on the metadata columns can take effect during the execution phase.

[0025] During the execution phase, the metadata columns in the metadata file corresponding to the data table are first read to obtain the metadata, and this metadata is then used to populate the scan filter conditions to make them effective. Next, the business data files in the data storage system are filtered according to the scan filter conditions populated with metadata to obtain the target business data file corresponding to the data table. Finally, the target business data columns in the target business data file are read column-wise to obtain the target business data. Thus, by reading the specified target business data columns in the target business data file using column-wise reading, the required target business data can be accurately and efficiently retrieved, while avoiding the additional overhead caused by reading unnecessary data. Furthermore, the target business data obtained through column-wise reading is vectorized data, which facilitates subsequent calculations. In summary, this application, for cases requiring scanning data tables stored in a columnar manner, converts the row-based scan operator in the execution plan into a vectorized scan operator covered with metadata columns. This allows for accurate filtering of the target business data file based on scan filter conditions. Moreover, by reading the target business data columns in the target business data file column-wise, the efficiency and performance of data processing can be improved, unnecessary overhead can be avoided, and computing resources of computer equipment can be saved.

[0026] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0027] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application. Figure 1 This is a schematic diagram of the architecture of the data processing system provided in an embodiment of this application; Figure 2 This is a schematic diagram of a scenario of the data processing method provided in an embodiment of this application; Figure 3 This is another scenario illustration of the data processing method provided in the embodiments of this application; Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the scanning filter provided in an embodiment of this application; Figure 6 This is another schematic diagram of the scanning filter provided in the embodiments of this application; Figure 7 This is another schematic flowchart of the data processing method provided in the embodiments of this application; Figure 8 This is another schematic flowchart of the data processing method provided in the embodiments of this application; Figure 9 This is another schematic flowchart of the data processing method provided in the embodiments of this application; Figure 10 This is a schematic diagram illustrating the execution process of a data manipulation statement provided in an embodiment of this application; Figure 11 This is a schematic diagram of the parsing layer provided in an embodiment of this application; Figure 12 This is a comparative schematic diagram of the row scan operator provided in the embodiments of this application and the vectorized scan operator of version V1; Figure 13 This is a flowchart illustrating a vectorized scan operator compatible with both version V1 and version V2, provided in this application embodiment. Figure 14 This is a flowchart illustrating the subquery merging process provided in an embodiment of this application; Figure 15This is a schematic diagram of two sub-queries provided in an embodiment of this application; Figure 16 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application; Figure 17 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 18 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0030] It is worth noting that in the specific embodiments of this application, data from the data storage system is involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the relevant parties is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain data from the data storage system, separate permission or consent from the relevant parties can be obtained through pop-up windows or redirection to a confirmation page. After obtaining the separate permission or consent from the relevant parties, the necessary data for the normal operation of the embodiment of this application can then be obtained.

[0031] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: 1) Data storage system: This generally refers to computer software systems that support the storage and management of data. For example, data storage systems can be implemented as databases, data warehouses, data lakes, integrated data lakehouses, and intelligent data warehouses. A database is a collection of data organized, stored, and managed according to a specific data model. A data warehouse is a subject-oriented, integrated, relatively stable collection of data reflecting historical changes, used to support management decisions. It accesses multiple data sources (such as log data, database data, application data, etc.) and integrates the data into a unified storage environment through processes such as extraction, transformation, and loading. A data lake is a repository that stores data in natural / raw formats, adopting a "store first, process later" approach. It can accommodate various formats and types of data, such as structured data, semi-structured data, and unstructured data, allowing for flexible processing of data in the data lake according to different analytical needs. A lake-warehouse integration connects the data warehouse and the data lake, combining the high performance and management capabilities of the data warehouse with the flexibility of the data lake. The underlying layer supports the coexistence of multiple data types and enables data sharing between them. The upper layer can access the data through a unified encapsulated interface, supporting real-time querying and analysis simultaneously. An intelligent data warehouse is centered around the data lake, forming a unified and continuous whole that integrates the data lake, data warehouse, and other data processing services (such as machine learning and log analysis). Of course, the above are merely examples of data storage systems and do not constitute a limitation on the embodiments of this application. The embodiments of this application can support data processing on other forms of data storage systems.

[0032] 2) Data Table: A data table is the basic structure used to organize and store data in a data storage system. It gathers related data together in a structured way, organizing the data according to rows and columns. In this embodiment, the data table includes not only business data related to actual business operations, but also metadata of the business data. Metadata provides descriptive information about the business data, making the management and querying of business data more efficient and flexible.

[0033] 3) Data Manipulation Language (DML): This is a type of Structured Query Language (SQL). Structured Query Language is used to manage and manipulate data storage systems, while Data Manipulation Language (DML) is used to perform operations such as inserting, querying, updating, or deleting data in the data storage system.

[0034] 4) Execution Plan: A series of operation steps (the specific actions of the operation steps can be called operators or computation operators) are formulated for executing data manipulation statements. It describes how to read data, what kind of processing to perform on the data, and the final return of the result.

[0035] 5) Columnar storage: This refers to organizing and storing data by columns, rather than by rows. Compared to row-based storage, columnar storage has significant advantages when processing big data and performing data analysis.

[0036] 6) Row-by-row scanning: refers to scanning data in the order of rows.

[0037] 7) Vectorized scanning: refers to scanning data in units of vectors (a set of consecutive data elements).

[0038] In today's digital age, the scale of big data is exploding, encompassing data from various fields such as the internet, finance, healthcare, and industry. This data is characterized by its massive volume, multiple sources, and heterogeneity, posing significant challenges to data storage and processing. To meet the demands of big data storage and analysis, various data storage and processing technologies have emerged.

[0039] Columnar storage has gradually become the mainstream method for big data storage due to its advantages in storage efficiency and query performance. However, current data computing engines (such as the Spark computing engine) usually adopt a row-based computing framework, which requires reorganizing column-stored data into row-stored data for computation. This column-to-row conversion process involves a large amount of data reorganization and memory operations, which consumes a lot of computing resources and time, resulting in poor data processing efficiency and performance.

[0040] In response to this, embodiments of this application provide a data processing method, apparatus, storage medium, computer equipment, and program product that can perform vectorized scanning on data tables using columnar storage, thereby improving the efficiency and performance of data processing.

[0041] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application. It includes a data storage system 150, a terminal 140, an Internet 130, a gateway 120, a server 110, etc.

[0042] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, forming a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0043] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0044] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0045] The data storage system 150 is managed by the server 110, meaning the server 110 performs data processing on the data storage system 150. For example, the server 110 receives data operation statements from the terminal 140 regarding the data storage system 150; the server 110 parses and processes the data operation statements to obtain a first execution plan. The first execution plan includes row-based scan operators for data tables stored in columnar format in the data storage system. The data tables include metadata columns and business data columns; the server 110 converts the row-based scan operators in the first execution plan into vectorized scan operators to obtain a second execution plan. The scan output columns of the vectorized scan operators include metadata columns and target business data columns. The metadata columns are used to construct the scan filtering conditions of the vectorized scan operators; the server 110 initiates the vectorized scan operators in the second execution plan, reads the metadata columns in the metadata file corresponding to the data table to obtain metadata, and fills the metadata into the scan filtering conditions; the server 110 filters the business data files in the data storage system according to the scan filtering conditions filled with metadata to obtain the target business data file corresponding to the data table, and performs columnar reading of the target business data columns in the target business data file to obtain the target business data. In some embodiments, the operations performed by the server 110 described above can be implemented by a data computing engine (such as the Spark computing engine). The data computing engine is a software component responsible for parsing, optimizing, and executing data operation statements to achieve data computation, and can also be called a data processing engine.

[0046] It is worth noting that, Figure 1 The architecture shown is merely an example and does not constitute a limitation on the embodiments of this application. For example, the data storage system 150 can be directly managed by the terminal 140 to achieve local data processing; or, for another example, the server 110 can generate data operation statements according to preset strategies to achieve automated data processing.

[0047] The embodiments of this application can be applied in various scenarios, such as Figure 2 The data warehouse scenario shown Figure 3 The data lake scenario shown below will be explained in detail.

[0048] 1) Data warehouse scenario.

[0049] A data warehouse is a subject-oriented, integrated, relatively stable collection of data that reflects historical changes, used to support management decisions. For example... Figure 2As shown, taking a company's data warehouse as an example, the data warehouse will connect to multiple data sources, such as internal sales data sources (including sales data from various channels), supply chain data sources (including supply chain data such as procurement data, supplier data, and logistics data), human resources data sources (including internal human resources data such as departmental structure and employee changes), and external data sources (including market research data and industry reports). For these multiple data sources, through processes such as extraction, transformation, and loading (ETL), the data from these data sources is integrated into the data warehouse and stored in a unified manner. Figure 2 The structured data shown refers to data that can be logically expressed using a two-dimensional table structure. Structured data stored in a data warehouse can be processed by a data processing engine. The operations performed by the data processing engine can be divided into three layers: parsing, optimization, and execution. The parsing layer parses and processes data operation statements to obtain an execution plan. The optimization layer optimizes the execution plan output by the parsing layer according to specific optimization rules. The execution layer executes the execution plan output by the optimization layer.

[0050] This application's embodiments introduce vectorized scan optimization rules in the optimization layer. For example, for the first execution plan output by the parsing layer, if the first execution plan includes a row-based scan operator for a data table (structured data) stored in columnar format in the data warehouse, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain a second execution plan. The scan output columns of the vectorized scan operator include a metadata column and a target business data column. The metadata column is used to construct the scan filtering conditions for the vectorized scan operator. Thus, when the execution layer executes the second execution plan output by the optimization layer, it reads the metadata column from the metadata file corresponding to the data table to obtain metadata, fills the metadata into the scan filtering conditions, filters the business data files in the data storage system according to the scan filtering conditions filled with metadata, obtains the target business data file corresponding to the data table, and performs columnar reading of the target business data column in the target business data file to obtain the target business data. For data tables stored in columnar format, the vectorized scan operator is significantly more efficient than the row-based scan operator. Therefore, this application's embodiments can greatly improve the efficiency and performance of data processing in the data warehouse.

[0051] 2) Data lake scenario.

[0052] A data lake is a repository that stores data in its natural / raw format, employing a "store first, process later" approach, and can accommodate data of various formats and types. For example... Figure 3As shown, taking a company's data lake as an example, the data lake will connect to multiple data sources, and data from these data sources can be directly written into the data lake. Figure 3 The diagram illustrates structured, semi-structured, and unstructured data stored in a data lake. Similarly, structured data stored in a data lake can be manipulated using a data computation engine. This data computation engine's operations can include three layers: a parsing layer, an optimization layer, and an execution layer. This embodiment of the application significantly improves the efficiency and performance of data processing in the data lake by introducing vectorized scanning optimization rules into the optimization layer.

[0053] Many business decisions in enterprise operations rely on the timely output of data. The embodiments of this application can accelerate data processing without upgrading hardware, whether it is a data warehouse, data lake or other data storage system, thereby improving the timeliness of business decisions and contributing to the healthy development of the enterprise's business.

[0054] It is worth noting that, Figure 1 The data processing system shown and Figures 2 to 3 The scenarios shown are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of computer technology and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0055] In this embodiment, the description will be from the perspective of a data processing device. Specifically, the data processing device can be integrated into a computer device that has a storage unit and is equipped with a microprocessor and has computing capabilities. The computer device can be a server and / or a terminal, and there is no limitation on this.

[0056] Please see Figure 4 , Figure 4 This is a flowchart illustrating the data processing method provided in the embodiments of this application. The data processing method can be implemented by a server and / or a terminal. Figure 4 The data processing methods shown include: Step 410: Receive data operation statements for the data storage system; Step 420: Parse and process the data operation statement to obtain the first execution plan. The first execution plan includes row scan operators for data tables that use columnar storage in the data storage system. The data table includes metadata columns and business data columns. Step 430: Convert the row-based scan operator in the first execution plan into a vectorized scan operator to obtain the second execution plan. The scan output columns of the vectorized scan operator include a metadata column and a target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator. Step 440: Start the vectorized scan operator in the second execution plan, read the metadata column in the metadata file corresponding to the data table to obtain the metadata, and fill the metadata into the scan filter conditions; Step 450: Filter the business data files in the data storage system according to the scanning filter conditions filled with metadata to obtain the target business data file corresponding to the data table, and perform columnar reading on the target business data columns in the target business data file to obtain the target business data. Steps 410-450 are described in detail below.

[0057] In step 410, data operation statements for the data storage system are received.

[0058] A data storage system generally refers to a computer software system that supports the storage and management of data, such as databases, data warehouses, data lakes, and integrated data storage and management systems. During the operation of a data storage system, it typically receives data manipulation statements. These statements are used to operate on the data within the system, including but not limited to data query statements (SELECT statements), data insertion statements (INSERT statements), data update statements (UPDATE statements), and data deletion statements (DELETE statements).

[0059] It is worth noting that in this embodiment, the data storage system stores data tables using columnar storage, such as data tables in Iceberg format. The data tables include metadata columns and business data columns. Metadata columns are special columns used to store metadata, which provides descriptive information about the file paths, partitions, etc., of the business data. The business data columns store business data related to the actual business operations. Furthermore, metadata and business data are stored separately; metadata is stored in metadata files, and business data is stored in business data files. For example, in Iceberg format, metadata files are manifest files, and business data files are data files. Columnar storage of the data tables mainly refers to the business data files using columnar storage (such as Parquet). Metadata files can use any storage format, such as columnar storage (such as Parquet) or binary storage (such as Avro), without limitation. To make it easier to understand, let's take a student grade table as an example to illustrate the meaning of columnar storage. A student grade table may include business data columns such as "name", "subject", and "grade". In columnar storage, the business data of the "name" column will be stored together, the business data of the "subject" column will be stored together, and the business data of the "grade" column will be stored together. This is similar to splitting each business data column in the student grade table separately and then storing the business data of each business data column in adjacent positions in a queue.

[0060] In step 420, the data operation statement is parsed to obtain a first execution plan. The first execution plan includes a row scan operator for a data table that uses columnar storage in the data storage system. The data table includes metadata columns and business data columns.

[0061] Here, the data manipulation statements can be processed by the data computation engine. The processing can be divided into a parsing layer, an optimization layer, and an execution layer. In the parsing layer, the data manipulation statements are parsed to obtain an execution plan. The execution plan is a description of the detailed operation steps for executing the data manipulation statements; the specific actions of these operation steps can be called computation operators. For ease of distinction, the execution plan obtained after parsing is called the first execution plan.

[0062] This application does not limit the specific implementation of the parsing process in its embodiments. For example, lexical analysis can be performed on the data operation statement first to break it down into independent lexical units (Tokens). These lexical units are the smallest meaningful units in the data operation statement, such as keywords (e.g., SELECT, FROM, WHERE), identifiers (e.g., table names, column names), operators (e.g., +, -, =), and constants (e.g., numbers, strings). Then, the multiple lexical units obtained from the lexical analysis are analyzed according to the syntax rules to construct a syntax tree. The syntax tree presents the syntactic structure of the data operation statement in a tree structure, with each node representing a syntactic element. The relationships between nodes reflect the hierarchical structure and logical relationships of the data operation statement. Then, semantic analysis is performed on the syntax tree, including verifying the existence of table names and column names, whether the data types match, and whether the permissions are sufficient. Semantic analysis is used to ensure the semantic correctness of the syntax tree. After semantic analysis of the syntax tree, an executable execution plan is generated based on the syntax tree, for example, mapping the nodes in the syntax tree to logical operators (computation operators) and determining the execution order of the logical operators. Computational operators include the Scan operator, Filter operator, Projection operator, and Join operator.

[0063] Since data computing engines typically employ a row-based computing framework, the first execution plan includes row-based scan operators for data tables stored in a column-based data storage system. Taking Spark as an example, row-based scan operators include SparkBatchQueryScan, which supports MergeOnRead tables, and SparkCopyOnWriteScan, which supports CopyOnWrite tables. The scan output columns of the row-based scan operator include metadata columns and target business data columns. The metadata columns are used to construct the scan filtering conditions for the row-based scan operator. The scan output columns refer to the specific columns that the row-based scan operator needs to read and pass to subsequent computation operators. The scan filtering conditions refer to the filtering conditions applied when the row-based scan operator is executed. It is worth noting that this application embodiment involves scan filtering conditions and filtering operators, which are different in meaning. Scan filtering conditions can be considered part of the row-based scan operator, while filtering operators are distinct from row-based scan operators and are typically executed after the row-based scan operator.

[0064] It is worth noting that the number of row scan operators in the first execution plan may be one or more, and there is no limit to this. For example, the first execution plan may include multiple subqueries, and each subquery may include a row scan operator.

[0065] In some embodiments, predicate pushdown can be used during the parsing process. Predicate pushdown is a strategy that pushes the filtering conditions (predicates) as close as possible to the data source for processing. This allows data filtering to occur during data scanning, avoiding unnecessary data processing. The aforementioned scanning filtering conditions can be constructed using predicate pushdown.

[0066] In some embodiments, column pruning can be used during the parsing process. Column pruning refers to selecting only the columns truly needed for processing based on the requirements of the data manipulation statement, while ignoring other irrelevant columns. If column pruning is used during parsing, the target business data columns mentioned above are usually a subset of all business data columns in the data table (of course, the possibility that the data manipulation statement requires all business data columns of the data table cannot be ruled out); if column pruning is not used during parsing, the target business data columns mentioned above are all business data columns in the data table.

[0067] In some embodiments, predicate pushdown and column pruning can also be employed in the optimization layer, i.e., as optimization rules in the optimization layer.

[0068] In step 430, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain the second execution plan. The scan output columns of the vectorized scan operator include a metadata column and a target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator.

[0069] If the first execution plan is executed directly at the execution layer, the row-based scan operator will read the business data file line by line (row-based reading). However, since the business data file is stored in columnar format, the row-based scan operator needs to reorganize the column-based data into row-based data for computation. This column-to-row conversion process involves a large amount of data reorganization and memory operations, which consumes a lot of computing resources and time. Furthermore, the row-by-row reading method reads all business data columns in the business data file, which may include unnecessary business data columns, resulting in unnecessary additional overhead.

[0070] Considering that row-based scan operators lead to poor data processing efficiency and performance, this embodiment converts the row-based scan operator in the first execution plan into a vectorized scan operator to obtain the second execution plan. However, most current vectorized scan operators are used for Hive tables and DataSource tables, which do not contain metadata columns. Therefore, current vectorized scan operators typically do not include code for reading metadata columns, meaning they do not cover the metadata columns. If the row-based scan operator in the first execution plan is converted to this vectorized scan operator, the metadata columns cannot be read correctly, leading to the failure of scan filtering conditions built based on the metadata columns, ultimately resulting in empty data. To address this, this embodiment improves upon the current vectorized scan operator by adding code for reading metadata columns. Since the vectorized scan operator includes code for reading metadata columns and code for columnar reading of business data columns, after converting the row-based scan operator in the first execution plan into a vectorized scan operator, the scan output columns of the vectorized scan operator will include metadata columns and target business data columns. The metadata columns are used to construct the scan filtering conditions of the vectorized scan operator.

[0071] In some embodiments, in addition to converting the row scan operator in the first execution plan into a vectorized scan operator, other optimization rules can also be implemented in the optimization layer, such as join operation optimization, index optimization, subquery optimization, aggregation and sorting optimization, etc., without limitation.

[0072] In step 440, the vectorized scan operator in the second execution plan is started to read the metadata columns in the metadata file corresponding to the data table to obtain the metadata, and then fill the metadata into the scan filter conditions.

[0073] Here, the execution plan output by the optimization layer is referred to as the second execution plan. The execution layer can execute the second execution plan. When the vectorized scan operator in the second execution plan is executed (equivalent to starting the vectorized scan operator in the second execution plan), the metadata columns in the metadata file corresponding to the data table are read to obtain the metadata, and the metadata is filled into the scan filter conditions.

[0074] It's worth noting that in data storage systems, the file paths of metadata files corresponding to data tables often follow a specific pattern (e.g., including the table name). Therefore, once the data table targeted by the vectorized scan operator is identified, the corresponding metadata file can be quickly located. Furthermore, the method for reading metadata columns from the metadata file depends on the file's storage format and is not limited. Taking an Avro file as an example, reading the metadata can be achieved by calling the Avro library.

[0075] In some embodiments, the number of vectorized scan operators includes multiple operators, and starting the vectorized scan operators in the second execution plan includes: starting multiple vectorized scan operators in the second execution plan serially, or starting multiple vectorized scan operators in the second execution plan in parallel.

[0076] When the second execution plan includes multiple vectorized scan operators, at the execution layer, these operators can be started sequentially (only one operator executes at a time) or in parallel (multiple operators execute simultaneously). For example, considering that parallel startup consumes more hardware resources than sequential startup, the availability of computer hardware resources (such as CPU availability and memory availability) can be obtained. When the hardware availability is less than or equal to the availability threshold, parallel startup may exhaust hardware resources, causing a sharp performance drop or even system crash. Therefore, starting multiple vectorized scan operators in the second execution plan sequentially ensures that each operator has sufficient hardware resources available during execution, avoiding excessive contention. When the hardware availability is greater than the availability threshold, it indicates that there are sufficient idle hardware resources available. Therefore, starting multiple vectorized scan operators in the second execution plan in parallel fully utilizes hardware resources and improves scanning efficiency.

[0077] In step 450, the business data files in the data storage system are filtered according to the scanning filtering conditions filled with metadata to obtain the target business data file corresponding to the data table, and the target business data columns in the target business data file are read in columnar form to obtain the target business data.

[0078] With metadata already populated in the scanning filter conditions, the business data files in the data storage system can be filtered to obtain the business data files corresponding to the data tables (referred to as target business data files for easy distinction). Then, the target business data columns in the target business data files are read column-wise to obtain vectorized target business data. On one hand, the target business data files themselves use column-wise storage, making column-wise reading more efficient than row-wise reading. On the other hand, column-wise reading only targets the target business data columns; other business data columns do not need to be read, effectively reducing workload. For example, if the target business data files include "Name," "Subject," "Score," and "Gender," and the target business data columns include "Name," "Subject," and "Score," then only the "Name," "Subject," and "Score" target business data columns need to be read column-wise; the "Gender" business data column does not need to be read.

[0079] It is worth noting that steps 440 and 450 above can be regarded as the execution process of the vectorized scan operator.

[0080] In some embodiments, the number of target business data columns includes multiple columns. Reading the target business data columns in the target business data file in a columnar manner to obtain the target business data includes: reading the multiple target business data columns in the target business data file in a columnar manner sequentially according to the priority of the multiple target business data columns to obtain the target business data.

[0081] Here, when there are multiple target business data files, ordered columnar reading can be achieved based on the priority of the multiple target business data columns. For example, high-priority target business data columns can be read first, followed by low-priority target business data columns. The priority of the target business data columns can be stored in metadata, set in real-time in the optimization layer, specified by data manipulation statements, or written during storage; there are no limitations on this.

[0082] Next, we will provide examples from various scenarios.

[0083] 1) Pre-filtering of data records. For example, if certain target business data columns (referred to as the first target business data column) are used to construct the scan filtering conditions of the vectorized scan operator, the priority of the first target business data column can be set to be higher than that of other target business data columns (referred to as the second target business data column) in the optimization layer. Then, in the execution layer, the first target business data column is read first and the data records are filtered. Then, the second target business data column is read from the filtered data records. This can reduce the workload of reading the second target business data column and avoid reading unnecessary data.

[0084] 2) Progressive result return. For example, if the priority of the target business data column reflects the criticality of the target business data column, then in the execution layer, the column can be read in an orderly manner according to the priority of multiple target business data columns, and the result of the column read can be returned in real time after each column read (e.g., displayed on the screen). In this way, data with higher criticality can be returned first, improving the timeliness of data processing and reducing perceived delay.

[0085] 3) Load control in resource-sensitive scenarios. For example, in resource-sensitive scenarios such as when hardware resources are scarce or high concurrency occurs, the execution layer can implement ordered columnar reading based on the priority of multiple target business data columns. This avoids resource contention caused by simultaneous columnar reading of multiple target business data columns, and can improve the stability and security of the columnar reading process.

[0086] In some embodiments, the metadata column and the target business data column are used to jointly construct the scanning filtering conditions of the vectorized scanning operator. The target business data is obtained by columnar reading of the target business data column in the target business data file, including: obtaining candidate business data by columnar reading of the target business data column in the target business data file, and filling the candidate business data into the scanning filtering conditions; filtering the data records in the target business data file according to the scanning filtering conditions filled with candidate business data to obtain a second target data record; and determining the candidate business data in the second target data record as the target business data.

[0087] Here, the metadata column and the target business data column can be used together to construct the scan filtering conditions of the vectorized scan operator. The part of the scan filtering conditions related to the target business data column can be obtained during the parsing and processing of data operation statements, such as "col1>1" in the data operation statement. In this case, the target business data column in the target business data file is read columnarly to obtain candidate business data. The candidate business data is then filled into the scan filtering conditions, and the data records in the target business data file are filtered according to the scan filtering conditions filled with candidate business data to obtain the second target data record. Here, the data record in the target business data file indicates the business data of each row (each business data row) in the target business data file, and the second target data record indicates the business data of a specific row in the target business data file across all business data columns. Then, the candidate business data in the second target data record is determined as the target business data.

[0088] like Figure 5 As shown, the target business data file consists of 5 rows and 5 columns. The target business data column is column 1. First, the first column of the target business data file is read to obtain candidate business data, which are the business data in columns 1 through 5 of the target business data file. Here, we assume the scan filter condition includes "col1>1" (i.e., the data in column 1 is greater than 1). After filling the scan filter condition with candidate business data, it is found that the business data in columns 1 through 3 of the target business data file is greater than 1. Therefore, the data records in rows 1 through 3 of the target business data file are used as the second target data records, and the business data in columns 1 through 3 of the target business data file is used as the final target business data. Taking a student grade table as an example to illustrate a practical application scenario, for example, if the target business data column is the "Grade" business data column, and the scan filter condition includes "Grade > 60 points", then the final target business data is all grades in the student grade table that are greater than 60 points.

[0089] The above method, in addition to reading the target business data columns in the target business data file in a columnar manner, also introduces filtering of data records. This is equivalent to performing double filtering of rows and columns on the target business data file, making the final target business data read more accurate and avoiding unnecessary interference from business data in subsequent processing.

[0090] In some embodiments, the target service data column includes a first target service data column and a second target service data column. The metadata column and the first target service data column are used to jointly construct the scanning filtering conditions of the vectorized scanning operator. The target service data is obtained by columnar reading of the target service data column in the target service data file, including: columnar reading of the first target service data column in the target service data file to obtain candidate service data, and filling the candidate service data into the scanning filtering conditions; filtering the data records in the target service data file according to the scanning filtering conditions filled with candidate service data to obtain a third target data record; determining the candidate service data in the third target data record as the first target service data; columnar reading of the second target service data column in the third target data record to obtain the second target service data; and determining the first target service data and the second target service data as the target service data.

[0091] Here, the target business data column can be divided into a first target business data column and a second target business data column. The first target business data column is used together with the metadata column to construct the scan filtering conditions of the vectorized scan operator, while the second target business data column does not participate in constructing the scan filtering conditions of the vectorized scan operator. In this case, the first target business data column in the target business data file is read column-wise to obtain candidate business data, and the candidate business data is filled into the scan filtering conditions. The data records in the target business data file are filtered according to the scan filtering conditions filled with candidate business data to obtain a third target data record. The third target data record indicates the business data of a specific row in the target business data file across all business data columns. The candidate business data in the third target data record (i.e., the business data in the first target business data column of the third target data record) is determined as the first target business data. The second target business data column in the third target data record is read column-wise to obtain the second target business data. The first target business data and the second target business data are determined as the target business data, that is, the target business data includes the first target business data and the second target business data.

[0092] like Figure 6As shown, the target business data file consists of 5 rows and 5 columns. The first target business data column is column 1, and the second target business data column is column 2. First, column 1 of the target business data file is read to obtain candidate business data, which consists of the business data in column 1 of rows 1 to 5. Here, it is assumed that the scan filter condition includes "col1>1" (i.e., the data in column 1 is greater than 1). After filling the scan filter condition with candidate business data, it is found that the business data in column 1 of rows 1 to 3 is greater than 1. Therefore, the data records in rows 1 to 3 are used as the third target data records. Then, the business data in column 1 of rows 1 to 3 is used as the first target business data. Simultaneously, column 2 of the data records in rows 1 to 3 is read to obtain the second target business data. The first and second target business data are combined to form the target business data. Taking a student grade table as an example to illustrate a practical application scenario, for instance, the first target business data column is the "grade" business data column, the second target business data column is the "name" business data column, and the scanning filter condition includes "grade > 60 points". Then the final target business data obtained includes all grades in the student grade table that are greater than 60 points, as well as the name of the student corresponding to each grade.

[0093] The above method uses scanning filtering conditions built on the first target business data column to filter the data records to obtain the third target data record. Then, the second target business data column in the third target data record is read in columnar form, so that the final target business data not only meets the scanning filtering conditions, but also contains the business data of all the required target business data columns, avoiding unnecessary business data from interfering with subsequent processing.

[0094] like Figure 4 As shown, this embodiment of the application receives data operation statements for a data storage system and triggers a three-stage processing phase: parsing, optimization, and execution. In the parsing phase, the data operation statements are parsed to form a first execution plan that includes a series of computational operators. These operators include a row-based scan operator for data tables stored in a columnar manner in the data storage system. The data table includes a metadata column for storing metadata and a business data column for storing business data. The metadata provides descriptive information about the business data, used to locate the business data.

[0095] During the optimization phase, considering that columnar storage data tables can be read in columnar format, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain the second execution plan. The scan output columns of the vectorized scan operator include metadata columns and target business data columns. The metadata columns are used to construct the scan filtering conditions of the vectorized scan operator. Since the scan output columns of the vectorized scan operator cover the metadata columns, it is ensured that the scan filtering conditions constructed based on the metadata columns can take effect during the execution phase.

[0096] During the execution phase, the metadata columns in the metadata file corresponding to the data table are first read to obtain the metadata, and this metadata is then used to populate the scan filter conditions to make them effective. Next, the business data files in the data storage system are filtered according to the scan filter conditions populated with metadata to obtain the target business data file corresponding to the data table. Finally, the target business data columns in the target business data file are read column-wise to obtain the target business data. Thus, by reading the specified target business data columns in the target business data file using column-wise reading, the required target business data can be accurately and efficiently retrieved, while avoiding the additional overhead of reading unnecessary data. Furthermore, the target business data obtained through column-wise reading is vectorized data, which facilitates subsequent calculations. In summary, this embodiment of the application, for cases requiring scanning data tables stored in columnar format, converts the row-based scan operator in the execution plan into a vectorized scan operator covered with metadata columns. This allows for accurate filtering of the target business data file based on scan filter conditions. Moreover, by reading the target business data columns in the target business data file column-wise, the efficiency and performance of data processing can be improved, unnecessary overhead can be avoided, and computing resources of computer equipment can be saved.

[0097] In some embodiments, refer to Figure 7 , Figure 7 This is a schematic diagram of an optional flow of the data processing method provided in an embodiment of this application, such as... Figure 7 As shown, Figure 4 The illustrated step 450 includes: Step 710: Filter the business data files in the data storage system according to the scanning filter conditions filled with the target file path to obtain the business data file with the target file path; Step 720: Determine the target business data file corresponding to the data table in the business data file of the target file path; Step 730: Read the target business data column from the target business data file to obtain the target business data.

[0098] Steps 710-720 are described in detail below.

[0099] In step 710, the business data files in the data storage system are filtered according to the scanning filtering conditions filled with the target file path to obtain the business data file with the target file path.

[0100] Here, the metadata columns include a file path column, which stores the file path of the business data file where the data record resides. Taking Iceberg format as an example, the file path column refers to the `_file` column. Based on this, the target file path is obtained by reading the file path column from the corresponding metadata file of the data table, and this target file path is then used to populate the scan filter conditions. Then, the business data files in the data storage system are filtered according to the scan filter conditions populated with the target file path to obtain the business data file with the target file path (i.e., the business data file whose file path is the same as the target file path).

[0101] In step 720, the target business data file corresponding to the data table is determined in the business data file of the target file path.

[0102] For example, if the metadata column does not include the partition column, or if there is only one business data file in the target file path, the business data file in the target file path can be directly determined as the target business data file corresponding to the data table.

[0103] In some embodiments, the metadata column further includes a partition column, and the metadata also includes target partition information read from the partition column in the metadata file. The business data file of the target file path includes multiple partition business data files located in different partitions. Determining the target business data file corresponding to the data table in the business data file of the target file path includes: filtering the multiple partition business data files according to the scan filtering conditions filled with target partition information to obtain the partition business data file located in the target partition corresponding to the target partition information; and determining the partition business data file located in the target partition as the target business data file corresponding to the data table.

[0104] Here, in addition to the file path column, the metadata column also includes a partition column, which stores the partition information of the business data file where the data records are located. Taking Iceberg format as an example, the partition column refers to the _partition column. It is worth noting that partitioning technology divides a data table into smaller, more manageable parts according to specific rules. Each partition can be seen as a subset of data, storing data that meets specific conditions. This application does not limit the rules on which the partitioning technology is based. For example, for businesses with obvious time characteristics (such as statistically analyzing sales performance on different dates), partitioning can be done based on date, with each partition storing business data for a specific date (such as a particular day or time period); for businesses with obvious geographical characteristics (such as statistically analyzing sales performance in different regions), partitioning can be done based on region, with each partition storing business data for a specific region; for businesses with obvious object characteristics (such as users) (such as object behavior analysis), partitioning can be done based on object attributes (such as age, gender, membership level, etc.), with each partition storing business data corresponding to specific attribute values. For example, the data can be partitioned based on the characteristics of the business data itself. For instance, a student grade table can be partitioned according to the score range, such as into three partitions: 0 to 59, 60 to 79, and 80 to 100.

[0105] Based on this, the partition column in the metadata file corresponding to the data table is read to obtain the target partition information, and this target partition information is then used to populate the scan filter conditions. As mentioned earlier, the business data files in the data storage system are filtered according to the scan filter conditions populated with the target file path to obtain the business data files at the target file path. Due to the introduction of partitioning technology, the business data files at the target file path will include multiple partition business data files located in different partitions. Therefore, here, multiple partition business data files are filtered according to the scan filter conditions populated with the target partition information to obtain the partition business data files located in the target partition corresponding to the target partition information. The partition business data files located in the target partition are then identified as the target business data files corresponding to the data table. Taking partitioning by date as an example, for instance, the business data files at the target file path include partition business data files located in the January 1st partition, partition business data files located in the January 2nd partition, and partition business data files located in the January 3rd partition. The target partition information is used to identify the January 3rd partition, so the January 3rd partition is the target partition, and the partition business data files located in the January 3rd partition are identified as the target business data files.

[0106] The above method, by introducing partitioning technology, enables more refined management of data tables. By quickly filtering the business data files located in the target partition using scan filtering conditions filled with target partition information, and identifying the business data files located in the target partition as the target business data files, the accuracy of filtering business data files can be improved, avoiding scanning business data files located in other partitions, and achieving further optimization of efficiency and performance.

[0107] In some embodiments, the metadata column further includes a data location column, and the metadata also includes the target data location read from the data location column in the metadata file. After determining the target business data file corresponding to the data table in the business data file of the target file path, the data processing method further includes: filtering the data records in the target business data file according to the scanning filter conditions filled with the target data location to obtain a first target data record; and performing columnar reading of the target business data column in the target business data file to obtain target business data, including: performing columnar reading of the target business data column in the first target data record to obtain target business data.

[0108] Here, in addition to the file path column, the metadata column also includes a data location column, which stores the specific data location of the data record in the business data file. Taking Iceberg format as an example, the data location column refers to the `_pos` column. Based on this, the target data location is obtained by reading the data location column from the metadata file corresponding to the data table, and this target data location is then filled into the scan filter conditions. After filtering the business data files in the data storage system according to the scan filter conditions filled with the target file path, the business data files at the target file path are obtained. After identifying the target business data file corresponding to the data table within the business data file at the target file path, the data records in the target business data file are filtered according to the scan filter conditions filled with the target data location to obtain the first target data record. The target business data is then obtained by columnar reading the target business data column in the first target data record.

[0109] The above method, based on the obtained target business data file, further filters the data records in the target business data file, thereby avoiding scanning unnecessary data records and achieving further optimization of efficiency and performance.

[0110] In some embodiments, the metadata column includes a file path column, a partition column, and a data location column; the metadata includes the target file path read from the file path column in the metadata file, the target partition information read from the partition column in the metadata file, and the target data location read from the data location column in the metadata file; filtering the business data files in the data storage system according to the scan filtering conditions filled with metadata to obtain the target business data file corresponding to the data table includes: filtering the business data files in the data storage system according to the scan filtering conditions filled with the target file path to obtain the business data file of the target file path, wherein the business data file of the target file path includes multiple partition business data files located in different partitions; filtering the business data files in the data storage system according to the scan filtering conditions filled with the target partition... The information scanning and filtering conditions filter multiple partition business data files to obtain the partition business data file located in the target partition corresponding to the target partition information. The partition business data file located in the target partition is then identified as the target business data file corresponding to the data table. After filtering the business data files in the data storage system according to the scanning and filtering conditions filled with metadata to obtain the target business data file corresponding to the data table, the process further includes: filtering the data records in the target business data file according to the scanning and filtering conditions filled with the target data location to obtain the first target data record; and performing columnar reading of the target business data columns in the target business data file to obtain the target business data, including: performing columnar reading of the target business data columns in the first target data record to obtain the target business data. Thus, through a three-layer filtering mechanism of target file path - target partition information - target data location, further optimization of efficiency and performance is achieved.

[0111] like Figure 7 As shown, this embodiment of the application accurately filters and obtains the target business data file based on the target file path. Compared with scanning all business data files indiscriminately in the data storage system, this embodiment of the application can greatly reduce the scanning range, improve the efficiency and performance of data processing, and at the same time ensure the accuracy of the final target business data.

[0112] In some embodiments, refer to Figure 8 , Figure 8 This is a schematic diagram of an optional flow of the data processing method provided in an embodiment of this application, such as... Figure 8 As shown, Figure 4 Step 430 shown includes: Step 810: Perform complexity analysis on the first execution plan to obtain the first complexity; Step 820: Perform complexity analysis on the data table to obtain the second complexity; Step 830: Perform prediction processing based on the first complexity and the second complexity to obtain the predicted execution time of the first execution plan; Step 840: When the predicted execution duration exceeds the duration threshold, convert the row scan operator in the first execution plan into a vectorized scan operator to obtain the second execution plan.

[0113] Steps 810-840 are described in detail below.

[0114] In step 810, the first execution plan is subjected to complexity analysis to obtain the first complexity.

[0115] For the first execution plan output by the parsing layer, since the individual computational operators and their execution levels are already defined, a complexity analysis is performed on the first execution plan to obtain the first complexity. For example, the first complexity can be the number of operators, the number of subqueries, the number of nesting levels, or a combination of these parameters (such as the average calculation result or the weighted summation result).

[0116] In step 820, the data table is subjected to complexity analysis to obtain the second complexity.

[0117] Here, a complexity analysis is performed on the data table to obtain a second complexity. For example, the second complexity could be the storage space occupied by the data table, the number of columns in the data table, or a combination of these parameters (such as a weighted sum). Parameters such as the storage space occupied by the data table and the number of columns in the data table can be obtained by triggering specific commands for querying; alternatively, the parsing process in step 420 will obtain the basic metadata of the data table, which includes parameters such as the storage space occupied by the data table and the number of columns in the data table.

[0118] In step 830, prediction processing is performed based on the first complexity and the second complexity to obtain the predicted execution time of the first execution plan.

[0119] Understandably, a higher first complexity indicates a more complex first execution plan, and a longer predicted execution time. A higher second complexity indicates a more complex data table targeted by the first execution plan, and a longer predicted execution time. For example, if the second complexity is the number of columns in the data table, the row scan operator in the first execution plan needs to scan all business data columns in the data table. The more columns in the data table, the greater the scanning workload of the row scan operator, and the longer the predicted execution time of the first execution plan.

[0120] Based on this, a prediction process can be performed using the first complexity and the second complexity to obtain the predicted execution time of the first execution plan. The predicted execution time is positively correlated with both the first and second complexities. For example, the first and second complexities can be weighted and summed to obtain the predicted execution time of the first execution plan; alternatively, a prediction model (such as a neural network model) can be pre-trained, and the first and second complexities can be input into the prediction model to obtain the predicted execution time of the first execution plan.

[0121] In some embodiments, the number of data tables includes multiple tables. Predictive processing is performed based on a first complexity and a second complexity to obtain the predicted execution time of the first execution plan, including: determining the proportion of operators corresponding to each of the multiple data tables in the first execution plan; performing predictive processing based on the first complexity and the second complexity corresponding to each of the multiple data tables to obtain the execution time of each data table; and using the proportion of operators corresponding to each of the multiple data tables as weights, performing a weighted summation of the execution times of each data table to obtain the predicted execution time of the first execution plan.

[0122] Here, when the first execution plan targets multiple data tables, we can first assume that the first execution plan targets only a single data table and determine the execution time for each data table. Taking the i-th data table as an example, we can perform prediction processing based on the first complexity and the second complexity corresponding to the i-th data table to obtain the execution time of the i-th data table. For example, we can perform a weighted summation of the first complexity and the second complexity corresponding to the i-th data table to obtain the execution time of the i-th data table; or, for example, we can pre-train a prediction model (such as a neural network model) and input the first complexity and the second complexity corresponding to the i-th data table into the prediction model, and the output of the prediction model is the execution time of the i-th data table. In addition, we also determine the proportion of operators corresponding to each of the multiple data tables in the first execution plan. For example, the proportion of operators corresponding to the i-th data table = the number of operators corresponding to the i-th data table / the number of operators in the first execution plan.

[0123] Finally, the proportion of operators corresponding to each of the multiple data tables is used as weights, and the execution time of each data table is weighted and summed to obtain the predicted execution time of the first execution plan.

[0124] The above method, when the first execution plan involves multiple data tables, comprehensively considers the second complexity of each data table and the proportion of each data table in the first execution plan, which can improve the accuracy of the predicted execution time. For example, although a certain data table has a high second complexity, if its proportion of operators in the first execution plan is small, then the impact of this data table on the predicted execution time is small. By using weighted summation, the overestimation of the impact of this data table on the predicted execution time can be avoided.

[0125] In step 840, when the predicted execution duration is greater than the duration threshold, the row scan operator in the first execution plan is converted into a vector scan operator to obtain the second execution plan.

[0126] When the predicted execution time is less than or equal to the duration threshold, it proves that the execution cost of the row scan operator is within an acceptable range. Considering that the process of converting the row scan operator into a vectorized scan operator also has a certain cost, the row scan operator in the first execution plan remains unchanged, and the row scan operator in the first execution plan is subsequently started to obtain the target business data. When the predicted execution time is greater than the duration threshold, it proves that the execution cost of the row scan operator is too high. Therefore, the row scan operator in the first execution plan is converted into a more efficient vectorized scan operator to obtain the second execution plan, and the actual execution time of the second execution plan is made to be less than or equal to the duration threshold as much as possible.

[0127] like Figure 8 As shown, this embodiment of the application predicts the execution time of the first execution plan, and when the predicted execution time exceeds a time threshold, converts the row-based scan operator in the first execution plan into a vectorized scan operator to obtain the second execution plan. On the one hand, this ensures the rationality and necessity of converting the scan operator; on the other hand, it minimizes the actual execution time of the execution layer, enabling timely responses to data manipulation statements.

[0128] In some embodiments, refer to Figure 9 , Figure 9 This is a schematic diagram of an optional flow of the data processing method provided in an embodiment of this application, such as... Figure 9 As shown, Figure 4 Following step 430, the following is also included: Step 910: Perform matching processing on multiple vectorized scan operators in the second execution plan; like Figure 9 As shown, Figure 4 Step 440 shown can be implemented as either step 920 or step 930: Step 920: When multiple vectorized scan operators in the second execution plan are successfully matched, the multiple vectorized scan operators in the second execution plan are merged into the target vectorized scan operator to obtain the third execution plan. The target vectorized scan operator in the third execution plan is started to read the metadata column in the metadata file corresponding to the data table to obtain the metadata, and the metadata is filled into the scan filter conditions. Step 930: When multiple vectorized scan operators in the second execution plan fail to match, start multiple vectorized scan operators in the second execution plan, read the metadata columns in the metadata file corresponding to the data table to obtain the metadata, and fill the metadata into the scan filter conditions.

[0129] Steps 910-930 are described in detail below.

[0130] In step 910, the multiple vectorized scan operators in the second execution plan are matched.

[0131] In the second execution plan, there may be a situation where one query is nested inside another query. The outer query is usually called the main query, and the nested query is usually called a subquery. Furthermore, in the second execution plan, there may also be a situation where a main query contains multiple nested subqueries, where each subquery includes a vectorized scan operator. In response to this, the embodiments of this application can implement subquery merging optimization rules at the optimization layer.

[0132] First, the multiple vectorized scan operators in the second execution plan are matched to determine whether there is a possibility of merging the multiple vectorized scan operators.

[0133] In step 920, when multiple vectorized scan operators in the second execution plan are successfully matched, the multiple vectorized scan operators in the second execution plan are merged into a target vectorized scan operator to obtain a third execution plan. The target vectorized scan operator in the third execution plan is started to read the metadata columns in the metadata file corresponding to the data table to obtain metadata, and the metadata is filled into the scan filter conditions.

[0134] When multiple vectorized scan operators in the second execution plan match successfully, they are merged into a target vectorized scan operator (equivalent to merging multiple subqueries in the second execution plan into a target subquery, which includes the target vectorized scan operator), resulting in the third execution plan. In subsequent execution layers, the target vectorized scan operator in the third execution plan is then activated.

[0135] It's worth noting that the target vectorized scan operator is one of multiple vectorized scan operators, replacing several other vectorized scan operators. Subsequently, the target vectorized scan operator in the third execution plan can be activated to read the metadata columns from the corresponding metadata file of the data table, obtain the metadata, and then populate the metadata into the scan filter conditions.

[0136] In some embodiments, the matching process for multiple vectorized scan operators in the second execution plan includes: determining that multiple vectorized scan operators are successfully matched when the data tables to be scanned by the multiple vectorized scan operators are the same, the scan output columns of the multiple vectorized scan operators have an inclusion relationship, and the scan filtering conditions of the multiple vectorized scan operators are the same.

[0137] This provides an example of matching multiple vectorized scan operators. A successful match is determined when multiple vectorized scan operators scan the same data table, their output columns have an inclusion relationship, and their scan filtering conditions are identical. It's worth noting that if multiple vectorized scan operators scan different data tables, their data sources differ, potentially leading to data corruption and logical errors after merging. Therefore, it's necessary to ensure that the data tables scanned by the multiple operators are the same. Ensuring an inclusion relationship in the output columns ensures that one operator's output column includes the output columns of other operators, allowing one operator to replace others. Ensuring identical scan filtering conditions ensures that the data range processed by the multiple operators is the same, preventing conflicts and confusion in scan filtering conditions after merging.

[0138] In some embodiments, merging multiple vectorized scan operators in the second execution plan into a target vectorized scan operator includes: when the scan output columns of multiple vectorized scan operators are obtained by column pruning, determining the scan output column with the largest range among the scan output columns of multiple vectorized scan operators, and determining the vectorized scan operator corresponding to the scan output column with the largest range as the target vectorized scan operator; when the scan output columns of multiple vectorized scan operators are not obtained by column pruning, determining any one vectorized scan operator as the target vectorized scan operator.

[0139] Here, when the scan output columns of multiple vectorized scan operators are obtained after column pruning, the scan output columns of different vectorized scan operators may differ. Therefore, the scan output column with the largest range among the scan output columns of multiple vectorized scan operators is determined, and the vectorized scan operator corresponding to the scan output column with the largest range is determined as the target vectorized scan operator. This ensures that the scan output column of the target vectorized scan operator can include (cover) the scan output columns of multiple vectorized scan operators before merging. When the scan output columns of multiple vectorized scan operators are not obtained after column pruning, the scan output columns of different vectorized scan operators are the same, that is, they include all business data columns of the data table. Therefore, any one vectorized scan operator can be determined as the target vectorized scan operator.

[0140] It's worth noting that with column pruning, the target business data columns in the scan output columns of the vectorized scan operator only contain the business data columns required by the data operation statements. This reduces the scan volume of the vectorized scan operator and improves scanning efficiency. Without column pruning, since the scan output columns of each vectorized scan operator include all business data columns of the data table, the probability of successful matching among multiple vectorized scan operators is increased, which is particularly advantageous when the number of merged vectorized scan operators is large. Based on this, the decision to use column pruning can be made according to the actual application scenario. For example, column pruning is used when the number of vectorized scan operators (or the number of subqueries) is less than or equal to a threshold; it is not used when the number of vectorized scan operators exceeds the threshold.

[0141] In some embodiments, multiple vectorized scan operators are located in different subqueries, and the subqueries also include computation operators other than vectorized scan operators. Matching multiple vectorized scan operators in the second execution plan includes: matching computation operators included in each of the multiple subqueries; when the computation operators included in each of the multiple subqueries are successfully matched, matching multiple vectorized scan operators included in each of the multiple subqueries; when multiple vectorized scan operators in the second execution plan are merged into a target vectorized scan operator, the data processing method further includes: merging the computation operators included in each of the multiple subqueries into a target computation operator.

[0142] Here, subqueries may also include computational operators other than vectorized scan operators. In this case, the computational operators included in each of the multiple subqueries can be matched first. If the computational operators included in each of the multiple subqueries match successfully, then the vectorized scan operators included in each of the multiple subqueries can be matched. If the computational operators included in each of the multiple subqueries fail to match, it proves that the multiple subqueries cannot be merged, so they can directly enter the execution layer. In this way, the necessity of matching the vectorized scan operators included in each of the multiple subqueries is ensured, avoiding unnecessary waste of resources.

[0143] When multiple subqueries' vectorized scan operators match successfully, they need to be merged into a target subquery. This process involves merging the vectorized scan operators of each subquery into a target vectorized scan operator, and also merging the computational operators of each subquery into a target computational operator. In other words, the target subquery includes both the target vectorized scan operator and the target computational operator. This achieves an effective merging of multiple subqueries, resulting in a more concise third execution plan.

[0144] In some embodiments, the computation operator includes a filtering operator. Matching the computation operators included in multiple subqueries includes: determining that the filtering operators included in multiple subqueries are successfully matched when the execution levels of the filtering operators included in multiple subqueries are the same; merging the computation operators included in multiple subqueries into a target computation operator includes: performing a union operation on the filtering operators included in multiple subqueries to obtain the target filtering operator.

[0145] Here, when each subquery includes a filtering operator, if the filtering operators included in multiple subqueries are at the same execution level, it is determined that the filtering operators included in the multiple subqueries have successfully matched. The execution level indicates the execution order (or execution position) of the filtering operators within the subqueries. It is worth noting that for vectorized scan operators, the scanning filtering conditions are used to filter the data to be scanned. If the scanning filtering conditions of different vectorized scan operators are different, the range of data to be scanned cannot be clearly defined after merging. Therefore, it is necessary to constrain the scanning filtering conditions of different vectorized scan operators to be the same. Filtering operators, on the other hand, are used to filter the already scanned data. Even if the filtering operators themselves are different (such as the filtering operators "col1>1" and "col2>1"), they can be merged by taking the union. Therefore, it is only necessary to constrain the execution levels of the filtering operators included in multiple subqueries to be the same, thereby ensuring consistency in execution order. Furthermore, the reason why it is not necessary to constrain the execution level to be the same when matching multiple vectorized scan operators is that the execution level of vectorized scan operators is usually the lowest level, and it can be assumed that multiple vectorized scan operators have the same execution level.

[0146] When merging multiple subqueries into a target subquery, in addition to merging the vectorized scan operators included in each subquery into a target vectorized scan operator, the filtering operators included in each subquery are also combined to obtain the target filtering operator. That is, the target subquery includes both the target vectorized scan operator and the target filtering operator. This achieves an effective merging of multiple subqueries, making the resulting third execution plan more concise. Subsequent executions can then integrate the various filters applied to the retrieved target business data into a single execution.

[0147] In some embodiments, the computational operator includes a non-filtering operator. Matching the computational operators included in multiple subqueries includes: determining that the non-filtering operators included in multiple subqueries are successfully matched when the execution level of the non-filtering operators included in multiple subqueries is the same, the operator type of the non-filtering operators included in multiple subqueries is the same, and the data columns targeted by the non-filtering operators included in multiple subqueries are the same; merging the computational operators included in multiple subqueries into a target computational operator includes: determining the non-filtering operator in any one of the subqueries as the target non-filtering operator.

[0148] Here, when each subquery includes a non-filtering operator, if the non-filtering operators included in multiple subqueries are at the same execution level, have the same operator type (e.g., all are aggregate operators), and target the same data column (e.g., all target data column col1), it proves that the operations performed by the non-filtering operators included in multiple subqueries are duplicated. Therefore, it is determined that the non-filtering operators included in multiple subqueries are successfully matched.

[0149] When merging multiple subqueries into a target subquery, in addition to merging the vectorized scan operators included in each subquery into a target vectorized scan operator, any non-filtered operator in any subquery is also identified as a target non-filtered operator. That is, the target subquery includes both the target vectorized scan operator and the target non-filtered operator. This achieves an effective merging of multiple subqueries, resulting in a more concise third execution plan and avoiding redundant operations in subsequent executions.

[0150] Of course, in some embodiments, the subquery may also include filtering operators, non-filtering operators, and vectorized scan operators at the same time.

[0151] In step 930, when multiple vectorized scan operators in the second execution plan fail to match, multiple vectorized scan operators in the second execution plan are started to read the metadata columns in the metadata file corresponding to the data table to obtain the metadata, and the metadata is filled into the scan filter conditions.

[0152] When multiple vectorized scan operators in the second execution plan fail to match, it proves that merging is impossible, so the process proceeds directly to the execution layer. In the execution layer, the multiple vectorized scan operators from the second execution plan are activated to complete the data scanning task.

[0153] like Figure 9 As shown, in this embodiment of the application, when the second execution plan includes multiple vectorized scan operators, the multiple vectorized scan operators in the second execution plan are matched, and when a match is successful, the multiple vectorized scan operators in the second execution plan are merged into a target vectorized scan operator to obtain the third execution plan. That is, the second execution plan is optimized into a more concise third execution plan in the optimization layer, avoiding repeated scanning of the data table, thereby improving the efficiency and performance of data processing.

[0154] The following describes an exemplary application of this application's embodiments in a real-world scenario, using a data storage system as a data lake, data tables within the data lake as Iceberg tables, and Spark as the data computing engine. It's worth noting that Iceberg is a table format. Iceberg tables include metadata columns and business data columns. Metadata columns are special columns used to store metadata, providing descriptive information about the file paths, partitions, etc., of the business data. The business data columns store business data related to the actual business operations. Furthermore, metadata and business data are stored separately; metadata resides in metadata files (manifest files), and business data resides in business data files (data files). In this embodiment, Iceberg tables use columnar storage to store business data; for example, the business data files are stored in Parquet format. Additionally, Spark is an in-memory computing engine that can integrate with data sources in formats such as Iceberg and Hive, providing query optimization capabilities.

[0155] In solutions provided by related technologies, row-wise scans of Iceberg tables are typically achieved by injecting Spark extensions provided by Iceberg into Spark. However, since the business data in Iceberg tables is usually stored in Parquet columnar format, row-wise scans require reorganizing the column-based data into row-based data. This column-to-row conversion process involves a large amount of data reorganization and memory operations, consuming significant computing resources and time, resulting in poor data processing efficiency and performance.

[0156] For ease of understanding, combined with Figure 10 This describes the process by which Spark scans the Iceberg table using SQL. This process can be divided into three layers: the parsing layer, the optimization layer, and the execution layer. The parsing layer parses the SQL to obtain an execution plan. The optimization layer optimizes the execution plan output by the parsing layer according to specific optimization rules. The execution layer executes the execution plan output by the optimization layer. As an example, [details omitted]. Figure 10 An execution plan consisting of three operators (represented by circles) is shown.

[0157] Apart from Figure 10 In addition, through Figure 11 This shows the execution plan obtained by parsing the SQL through the parsing layer, which will be combined with... Figure 11 Provide an example. Figure 11 The SQL shown is as follows: “select col1 from TABLE where col3>2 join ( select col2 from TABLE2 join TABLE on TABLE2.col3=TABLE.col3 )t on TABLE.col2=t.col2” Figure 11 The operators in the execution plan shown are, from bottom to top, as follows: 1) Operator 6: Scan operator, specifically "Relation: TABLE, lcebergTableOutput: _file, col3", which means scanning the _file column and the col3 column from the data table TABLE in Iceberg format.

[0158] 2) Operator 5: Scan operator, specifically "Relation: TABLE2, lcebergTableOutput: _file, col3", which means scanning the _file column and col3 column from the data table TABLE2 in Iceberg format.

[0159] 3) Operator 4: Join operator, specifically "Join: left.col3 = right.col3", which means joining data table TABLE2 and data table TABLE through column col3. After operator 4, column col2 is also selected from the join result to serve as a temporary data table t. Figure 11 The execution plan is not shown.

[0160] 4) Operator 3: Scan operator, specifically "Relation: TABLE, lcebergTableOutput: _file, col1, col2, col3", which means scanning the _file, col1, col2, and col3 columns from the data table TABLE in Iceberg format.

[0161] 5) Operator 2: Filter operator, specifically "Filter: col3>2", which means to filter out data records from the data table TABLE whose value in column col3 is greater than 2.

[0162] 6) Operator 1: Join operator, specifically "Join: left.col2 = right.col2", which means joining data table TABLE and data table t through column col2. After operator 1, column col1 is also selected from the join result. Figure 11(The execution plan is not shown). It is worth noting that columns col1 to col3 mentioned above refer to the business data columns.

[0163] After the SQL is parsed and processed by the parsing layer, the basic metadata of the Iceberg table is obtained, such as the storage path of the Iceberg table (which can be regarded as the root path), the column structure and column types of the Iceberg table, etc. It is worth noting that... Figure 11 The scan operators shown in the execution plans are all row-based scan operators, such as SparkBatchQueryScan which supports MergeOnRead tables, or SparkCopyOnWriteScan which supports CopyOnWrite tables. The output columns of row-based scan operators include the `_file` metadata column (corresponding to the file path column mentioned above). The metadata stored in the `_file` metadata column is used to mark the target file path (the specific file path under the root path) to be read by the current task. Furthermore, the `_file` metadata column is used to construct scan filtering conditions to filter out the target business data files based on the target file path during the scan process. Of course, in addition to the `_file` metadata column, there may be more metadata columns, such as the `_partition` metadata column (corresponding to the partition column mentioned above) and the `_pos` metadata column (corresponding to the data location column mentioned above). The following explanation will only use the `_file` metadata column as an example.

[0164] Given that the business data files are stored in Parquet format, the row-based scan operator requires reorganizing the column-based data into row-based data during execution. This column-to-row conversion involves a large amount of data reorganization and memory operations, consuming significant computing resources and time, resulting in poor data processing efficiency and performance. Therefore, this embodiment of the application implements vectorized scan optimization rules in the optimization layer for the execution plan output by the parsing layer. These rules are used to convert the row-based scan operator in the execution plan into a vectorized scan operator, thereby improving the efficiency and performance of data processing.

[0165] However, most current vectorized scan operators (hereinafter referred to as V1 version vectorized scan operators) are used for Hive tables and datasource tables. These table formats do not contain a `_file` metadata column. Therefore, V1 version vectorized scan operators typically do not include code to read the `_file` metadata column, meaning they do not cover the `_file` metadata column. Based on the preceding text, the `_file` metadata column is used to construct scan filtering conditions. If the scan operator fails to correctly read the `_file` metadata column, the scan filtering conditions will fail, leading to the inability to successfully find the target business data file. For example... Figure 12As shown, the row-based scan operator's output columns include the `_file` metadata column and the target business data column (such as business data column 1, business data column 2, etc.). Therefore, scan filtering conditions built based on the `_file` metadata column can take effect. For example, by filling the scan filtering conditions with the metadata read from the `_file` metadata column, we get "_file is in (PATH1, PATH2)". PATH1 and PATH2 represent the target file paths. We can then find the corresponding target business data file based on PATH1 and PATH2 and read the target business data from it. However, if the row-based scan operator is converted to the V1 version vectorized scan operator, the V1 version vectorized scan operator cannot correctly read the `_file` metadata column. This will cause the scan filtering conditions built based on the `_file` metadata column to fail, ultimately resulting in empty data.

[0166] Therefore, this application embodiment improves the vectorized scan operator of version V1 by adding code to read the _file metadata column, causing it to override the _file metadata column, resulting in the vectorized scan operator of version V2. Furthermore, this application embodiment is compatible with both the vectorized scan operator of version V1 and version V2. Figure 13 As shown, after converting the row-based scan operator to the vectorized scan operator, the first step is to determine whether the scan output column of the vectorized scan operator includes the _file metadata column. If the scan output column of the vectorized scan operator includes the _file metadata column, it indicates that it is the V2 version of the vectorized scan operator. Therefore, the V2 version of the vectorized scan operator is executed normally, and the target business data is finally obtained. If the scan output column of the vectorized scan operator does not include the _file metadata column, it indicates that it is the V1 version of the vectorized scan operator. Since the V1 version of the vectorized scan operator can only obtain empty data after execution, the V1 version of the vectorized scan operator is converted back to the row-based scan operator. In this way, regardless of whether the V1 version or the V2 version of the vectorized scan operator is used in the actual application scenario, the target business data can be read correctly, improving compatibility.

[0167] Furthermore, the parsing phase involves column pruning, a scheme to filter business data columns. Specifically, it involves pruning the target business data columns relevant to the SQL query from all business data columns in the Iceberg table. In vectorized scan scenarios, since only the target business data columns need to be read column-wise, extra data entering memory for computation is avoided, effectively improving efficiency and performance. However, when an execution plan contains multiple subqueries targeting the same Iceberg table, these subqueries may scan different columns, preventing them from being merged and forcing them to be executed individually, leading to inefficiency and performance degradation.

[0168] To address this, in addition to implementing vectorized scanning optimization rules at the optimization layer, the embodiments of this application also implement subquery merging optimization rules at the optimization layer. For example... Figure 14 As shown, taking the case of two subqueries as an example, the optimization rules for subquery merging can include the following three steps: 1) Determine if the computational operators in the two subqueries, excluding the vectorized scan operator, match successfully. If the match is successful, proceed to step 2); if the match fails, terminate the subquery merging. Different matching rules can be applied to different types of computational operators. For example, for non-filtering operators, a match is considered successful only if the non-filtering operators included in the two subqueries are at the same execution level, have the same operator type, and target the same data columns. As another example, for filtering operators (different from the scan filtering conditions within the vectorized scan operator), since the filtering operators included in the two subqueries support union, a match is determined to be successful if the filtering operators included in the two subqueries are at the same execution level.

[0169] 2) Determine if there is an inclusion relationship between the scan output columns of the vectorized scan operators included in the two subqueries. If there is, proceed to step 3); otherwise, terminate the subquery merging.

[0170] 3) Determine if the scan filtering conditions of the vectorized scan operators included in the two subqueries are the same. If they are the same (proving that the vectorized scan operators included in the two subqueries match successfully), then merge the two subqueries into a target subquery; otherwise, terminate the subquery merging. The target subquery includes a target vectorized scan operator, a target filtering operator, and a target non-filtering operator. The target vectorized scan operator is the vectorized scan operator with the largest range of scanned output columns in the two subqueries. The target filtering operator is obtained by taking the union of the filtering operators included in the two subqueries. The target non-filtering operator is the non-filtering operator included in either of the two subqueries.

[0171] As an example, embodiments of this application provide, for instance, the following: Figure 15The diagram shows two subqueries. In subquery 1, the aggregation operator 1 is "Aggregate: col1", the filter operator 1 is "Filter:col1>1", and the vectorized scan operator 1 is "Scan:IcebergOutput:col1Filter:col1>1". In subquery 2, the aggregation operator 2 is "Aggregate: col1", the filter operator 2 is "Filter:col2>1", and the vectorized scan operator 2 is "Scan:IcebergOutput:col1,col2Filter:col1>1". Then, following the execution level from highest to lowest, the algorithm sequentially checks whether the computational operators at each execution level in the two subqueries match. First, it checks whether aggregate operator 1 and aggregate operator 2 match. Since aggregate operator 1 and aggregate operator 2 are at the same execution level, both are aggregate operators, and both target the data column col1, they are confirmed to match successfully. Next, it checks whether filter operator 1 and filter operator 2 match. Since filter operator 1 and filter operator 2 are at the same execution level, they are confirmed to match successfully. Finally, it checks the vectorized scan... The matching of operator 1 and vectorized scan operator 2 is determined by the fact that the scan output columns "col1, col2" of vectorized scan operator 2 include the scan output column "col1" of vectorized scan operator 1, and the scan filter condition "col1>1" of vectorized scan operator 2 is the same as that of vectorized scan operator 1. Therefore, vectorized scan operator 1 and vectorized scan operator 2 are determined to be a successful match. Subquery 1 and subquery 2 are merged into a target subquery. The target subquery includes, in descending order of execution level, the aggregation operator "Aggregate: col1", the filter operator "Filter:col1>1 ∪ col2>1", and the vectorized scan operator "Scan:IcebergOutput:col1, col2Filter:col1>1", where "∪" represents the union. In this way, replacing the original two subqueries with a single target subquery avoids repeated scans of the same Iceberg table, improving the efficiency and performance of data processing.

[0172] In some embodiments, column pruning can be canceled during the parsing phase. In this case, the scan output columns of the vectorized scan operator will include all business data columns of the Iceberg table. The scan output columns of the vectorized scan operators included in different subqueries will necessarily have an inclusion relationship, thereby increasing the probability of successful subquery merging. The above method can bring efficiency and performance improvements when the number of merged subqueries is large.

[0173] This application's embodiments, by implementing vectorized scanning optimization rules at the optimization layer, can support Spark's vectorized scanning of Iceberg tables and can be applied to data processing scenarios such as Merge, Delete, and Update. By implementing subquery merging optimization rules at the optimization layer, better efficiency and performance can be achieved in application scenarios with multiple subqueries. The inventors tested the data using 1TB of data and a cluster test with 128 Compute Units (CUs) based on the Transaction Processing Performance Council Decision Support Benchmark (TPC-DS) big data standard performance. After applying this application's embodiments, the time taken for Q09 (the standard query defined by TPC-DS) decreased from 166s to 117s, a reduction of 29.5%.

[0174] To facilitate better implementation of the data processing method provided in the embodiments of this application, the embodiments of this application also provide an apparatus based on the above-described data processing method. The meanings of the terms used are the same as in the data processing method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0175] Please see Figure 16 , Figure 16 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device is applied to a computer device and may include a receiving module 1610, a parsing module 1620, an optimization module 1630, and an execution module 1640, wherein: The receiving module 1610 is used to receive data operation statements for the data storage system; Parsing module 1620 is used to parse and process data operation statements to obtain a first execution plan. The first execution plan includes a row scan operator for a data table that uses columnar storage in the data storage system. The data table includes metadata columns and business data columns. Optimization module 1630 is used to convert the row-based scan operator in the first execution plan into a vectorized scan operator to obtain the second execution plan. The scan output columns of the vectorized scan operator include a metadata column and a target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator. The execution module 1640 is used to start the vectorized scan operator in the second execution plan, read the metadata column in the metadata file corresponding to the data table to obtain the metadata, and fill the metadata into the scan filter conditions; The execution module 1640 is also used to filter the business data files in the data storage system according to the scanning filtering conditions filled with metadata, obtain the target business data file corresponding to the data table, and perform columnar reading of the target business data columns in the target business data file to obtain the target business data.

[0176] In some embodiments, the metadata column includes a file path column, and the metadata includes the target file path read from the file path column in the metadata file. The execution module 1640 is further configured to: The business data files in the data storage system are filtered according to the scanning filter conditions filled with the target file path to obtain the business data file with the target file path; In the target file path, determine the target business data file corresponding to the data table.

[0177] In some embodiments, the metadata column further includes a partition column, and the metadata also includes target partition information read from the partition column in the metadata file. The business data file of the target file path includes multiple partition business data files located in different partitions. The execution module 1640 is further configured to: Based on the scanning filtering conditions filled with target partition information, multiple partition business data files are filtered to obtain the partition business data file located in the target partition corresponding to the target partition information; The partition business data file located in the target partition is identified as the target business data file corresponding to the data table.

[0178] In some embodiments, the metadata column further includes a data location column, and the metadata also includes the target data location read from the data location column in the metadata file. The execution module 1640 is further configured to: The data records in the target business data file are filtered according to the scanning filter conditions filled with the target data location to obtain the first target data record; The target business data is obtained by performing columnar reading on the target business data column in the first target data record.

[0179] In some embodiments, the optimization module 1630 is further configured to: The complexity of the first execution plan is obtained by performing a complexity analysis. Complexity analysis of the data table yields the second complexity. Based on the first complexity and the second complexity, a prediction process is performed to obtain the predicted execution time of the first execution plan; When the predicted execution duration exceeds the duration threshold, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain the second execution plan.

[0180] In some embodiments, the number of data tables includes multiple tables, and the optimization module 1630 is further configured to: Determine the proportion of operators corresponding to each of the multiple data tables in the first execution plan; Based on the first complexity and the second complexity corresponding to each of the multiple data tables, prediction processing is performed to obtain the execution time of each of the multiple data tables. Using the proportion of operators corresponding to multiple data tables as weights, the execution time of each data table is weighted and summed to obtain the predicted execution time of the first execution plan.

[0181] In some embodiments, the metadata column and the target business data column are used together to construct the scan filtering conditions of the vectorized scan operator. The execution module 1640 is further used for: The target business data column in the target business data file is read column-wise to obtain candidate business data, and the candidate business data is then filled into the scan filter conditions. The data records in the target business data file are filtered according to the scanning and filtering conditions filled with candidate business data to obtain the second target data record; The candidate business data in the second target data record is determined as the target business data.

[0182] In some embodiments, the target business data column includes a first target business data column and a second target business data column. The metadata column and the first target business data column are used to jointly construct the scan filtering conditions of the vectorized scan operator. The execution module 1640 is further used for: The first target business data column in the target business data file is read in columnar fashion to obtain candidate business data, and the candidate business data is then filled into the scan filtering conditions. The data records in the target business data file are filtered according to the scanning and filtering conditions filled with candidate business data to obtain the third target data record; The candidate business data in the third target data record is identified as the first target business data; The second target business data column in the third target data record is read in columnar format to obtain the second target business data. The first target business data and the second target business data are identified as target business data.

[0183] In some embodiments, the number of vectorized scan operators includes multiple ones, and the optimization module 1630 is further used for: Perform matching processing on multiple vectorized scan operators in the second execution plan; When multiple vectorized scan operators in the second execution plan are successfully matched, the multiple vectorized scan operators in the second execution plan are merged into the target vectorized scan operator to obtain the third execution plan, and the target vectorized scan operator in the third execution plan is started; When multiple vectorized scan operators in the second execution plan fail to match, the multiple vectorized scan operators in the second execution plan are started.

[0184] In some embodiments, the optimization module 1630 is further configured to: When multiple vectorized scan operators scan the same data table, the scan output columns of multiple vectorized scan operators have an inclusion relationship, and the scan filtering conditions of multiple vectorized scan operators are the same, it is determined that multiple vectorized scan operators have successfully matched.

[0185] In some embodiments, the optimization module 1630 is further configured to: When the scan output columns of multiple vectorized scan operators are obtained by column clipping, the scan output column with the largest range is determined among the scan output columns of multiple vectorized scan operators, and the vectorized scan operator corresponding to the scan output column with the largest range is determined as the target vectorized scan operator. When the scan output columns of multiple vectorized scan operators are obtained without column clipping, any one of the vectorized scan operators is determined as the target vectorized scan operator.

[0186] In some embodiments, multiple vectorized scan operators are located in different subqueries, and the subqueries also include computation operators other than the vectorized scan operators. The optimization module 1630 is further configured to: Perform matching processing on the calculated operators included in multiple subqueries; When the computed operators included in multiple subqueries are successfully matched, the vectorized scan operators included in the multiple subqueries are matched. The computed operators included in multiple subqueries are merged into the target computed operator.

[0187] In some embodiments, the computation operator includes a filtering operator, and the optimization module 1630 is further configured to: When multiple subqueries contain filtering operators at the same execution level, it is determined that the filtering operators contained in the multiple subqueries are successfully matched. The target filter operator is obtained by performing a union operation on the filter operators included in multiple subqueries.

[0188] In some embodiments, the computation operator includes a non-filtering operator, and the optimization module 1630 is further configured to: When multiple subqueries include non-filtered operators at the same execution level, have the same operator type, and target the same data columns, the non-filtered operators included in the multiple subqueries are considered to be successfully matched. The non-filtered operator in any subquery is identified as the target non-filtered operator.

[0189] For details on the implementation of each of the above modules, please refer to the previous examples, which will not be repeated here.

[0190] Reference Figure 17 , Figure 17 To implement the structural block diagram of a portion of the terminal 140 in this application embodiment, the terminal 140 includes: a radio frequency (RF) circuit 1710, a memory 1717, an input unit 1730, a display unit 1740, a sensor 1750, an audio circuit 1760, a wireless fidelity (WiFi) module 1770, a processor 1780, and a power supply 1790, among other components. Those skilled in the art will understand that... Figure 17 The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0191] RF circuit 1710 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with processor 1780; in addition, it transmits uplink data to the base station.

[0192] The memory 1717 can be used to store software programs and modules. The processor 1780 executes various terminal functions and data processing by running the software programs and modules stored in the memory 1717.

[0193] The input unit 1730 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 1730 may include a touch panel 1731 and other input devices 1732.

[0194] The display unit 1740 can be used to display input or provided information, as well as various menus of the terminal. The display unit 1740 may include a display panel 1741.

[0195] Audio circuitry 1760, speaker 1761, and microphone 1762 provide an audio interface.

[0196] In this embodiment, the processor 1780 included in the terminal 140 can execute the data processing method of the previous embodiment.

[0197] The terminal 140 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0198] Figure 18 This is a partial structural block diagram of a server 110 implementing an embodiment of this application. The server 110 can vary significantly due to different configurations or performance characteristics, and may include one or more Central Processing Units (CPUs) 1822 (e.g., one or more processors) and a memory 1832, and one or more storage media 1830 (e.g., one or more mass storage devices) for storing application programs 1842 or data 1844. The memory 1832 and storage media 1830 may be temporary or persistent storage. The program stored in the storage media 1830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 110. Furthermore, the CPU 1822 may be configured to communicate with the storage media 1830 and execute the series of instruction operations stored in the storage media 1830 on the server 110.

[0199] Server 110 may also include one or more power supplies 1826, one or more wired or wireless network interfaces 1850, one or more input / output interfaces 1858, and / or one or more operating systems 1841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0200] The central processing unit 1822 in server 110 can be used to execute the data processing method of the embodiments of this application.

[0201] This application also provides a computer-readable storage medium for storing program code for executing the data processing methods of the foregoing embodiments.

[0202] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the data processing method described above.

[0203] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0204] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0205] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0209] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0211] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0212] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A data processing method, characterized in that, include: Receive data manipulation statements for the data storage system; The data operation statement is parsed and processed to obtain a first execution plan. The first execution plan includes a row scan operator for a data table that uses columnar storage in the data storage system. The data table includes metadata columns and business data columns. The row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain a second execution plan. The scan output column of the vectorized scan operator includes the metadata column and the target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator. The vectorized scan operator in the second execution plan is started to read the metadata column in the metadata file corresponding to the data table to obtain the metadata, and the metadata is then filled into the scan filter conditions; The business data files in the data storage system are filtered according to the scanning filtering conditions filled with the metadata to obtain the target business data file corresponding to the data table, and the target business data columns in the target business data file are read in a columnar manner to obtain the target business data.

2. The data processing method according to claim 1, characterized in that, The metadata column includes a file path column, and the metadata includes the target file path read from the file path column in the metadata file. The step of filtering the business data files in the data storage system according to the scanning filtering conditions filled with the metadata to obtain the target business data file corresponding to the data table includes: The business data files in the data storage system are filtered according to the scanning and filtering conditions filled with the target file path to obtain the business data file of the target file path; The target business data file corresponding to the data table is determined in the business data file of the target file path.

3. The data processing method according to claim 2, characterized in that, The metadata column also includes a partition column, and the metadata also includes target partition information read from the partition column in the metadata file. The business data file of the target file path includes multiple partition business data files located in different partitions. Determining the target business data file corresponding to the data table in the business data file of the target file path includes: The multiple partition service data files are filtered according to the scanning filtering conditions filled with the target partition information to obtain the partition service data file located in the target partition corresponding to the target partition information; The partition business data file located in the target partition is determined as the target business data file corresponding to the data table.

4. The data processing method according to claim 2, characterized in that, The metadata column also includes a data location column, and the metadata also includes the target data location read from the data location column in the metadata file. After determining the target business data file corresponding to the data table in the business data file of the target file path, the data processing method further includes: The data records in the target business data file are filtered according to the scanning filtering conditions filled with the target data location to obtain the first target data record; The step of obtaining target business data by columnar reading of the target business data columns in the target business data file includes: The target business data is obtained by performing columnar reading on the target business data column in the first target data record.

5. The data processing method according to claim 1, characterized in that, The step of converting the row-based scan operator in the first execution plan into a vectorized scan operator to obtain the second execution plan includes: The first execution plan is subjected to complexity analysis to obtain the first complexity. The data table is subjected to complexity analysis to obtain the second complexity. Based on the first complexity and the second complexity, a prediction process is performed to obtain the predicted execution time of the first execution plan; When the predicted execution duration exceeds the duration threshold, the row-based scan operator in the first execution plan is converted into a vectorized scan operator to obtain the second execution plan.

6. The data processing method according to claim 5, characterized in that, The number of data tables includes multiple tables. The step of performing prediction processing based on the first complexity and the second complexity to obtain the predicted execution time of the first execution plan includes: Determine the proportion of operators corresponding to each of the multiple data tables in the first execution plan; Based on the first complexity and the second complexity corresponding to the multiple data tables, a prediction process is performed to obtain the execution time of the data tables corresponding to the multiple data tables. Using the proportion of operators corresponding to the multiple data tables as weights, the execution time of the data tables corresponding to the multiple data tables is weighted and summed to obtain the predicted execution time of the first execution plan.

7. The data processing method according to claim 1, characterized in that, The metadata column and the target business data column are used together to construct the scanning filtering conditions of the vectorized scanning operator. The step of performing columnar reading of the target business data column in the target business data file to obtain the target business data includes: Candidate business data is obtained by performing columnar reading on the target business data column in the target business data file, and the candidate business data is then filled into the scanning filter conditions. The data records in the target business data file are filtered according to the scanning and filtering conditions filled with the candidate business data to obtain the second target data record; The candidate business data in the second target data record is determined as the target business data.

8. The data processing method according to claim 1, characterized in that, The target business data column includes a first target business data column and a second target business data column. The metadata column and the first target business data column are used together to construct the scanning filtering conditions of the vectorized scanning operator. The step of reading the target business data column in the target business data file in a columnar manner to obtain the target business data includes: The first target business data column in the target business data file is read in a columnar manner to obtain candidate business data, and the candidate business data is then filled into the scanning filter conditions. The data records in the target business data file are filtered according to the scanning and filtering conditions filled with the candidate business data to obtain the third target data record; The candidate service data in the third target data record is determined as the first target service data; The second target business data column in the third target data record is read in a columnar manner to obtain the second target business data; The first target business data and the second target business data are identified as target business data.

9. The data processing method according to any one of claims 1 to 8, characterized in that, The number of vectorized scan operators includes multiple types. After converting the row-based scan operators in the first execution plan into vectorized scan operators to obtain the second execution plan, the data processing method further includes: The matching process is performed on multiple vectorized scan operators in the second execution plan; The initiation of the vectorized scan operator in the second execution plan includes: When the multiple vectorized scan operators in the second execution plan are successfully matched, the multiple vectorized scan operators in the second execution plan are merged into a target vectorized scan operator to obtain a third execution plan, and the target vectorized scan operator in the third execution plan is started; The initiation of the vectorized scan operator in the second execution plan includes: When the plurality of vectorized scan operators in the second execution plan fail to match, the plurality of vectorized scan operators in the second execution plan are started.

10. The data processing method according to claim 9, characterized in that, The matching process for multiple vectorized scan operators in the second execution plan includes: When the data tables to be scanned by the multiple vectorized scan operators are the same, the scan output columns of the multiple vectorized scan operators have an inclusion relationship, and the scan filtering conditions of the multiple vectorized scan operators are the same, it is determined that the multiple vectorized scan operators have successfully matched.

11. The data processing method according to claim 10, characterized in that, The step of merging the plurality of vectorized scan operators in the second execution plan into a target vectorized scan operator includes: When the scan output columns of the plurality of vectorized scan operators are obtained by column clipping, the scan output column with the largest range is determined among the scan output columns of the plurality of vectorized scan operators, and the vectorized scan operator corresponding to the scan output column with the largest range is determined as the target vectorized scan operator; When the scan output columns of the plurality of vectorized scan operators are obtained without column clipping, any one of the vectorized scan operators is determined as the target vectorized scan operator.

12. The data processing method according to claim 9, characterized in that, The multiple vectorized scan operators are located in different subqueries, and the subqueries also include computation operators other than the vectorized scan operators. The matching process for the multiple vectorized scan operators in the second execution plan includes: Perform matching processing on the calculated operators included in multiple subqueries; When the computational operators included in the multiple subqueries are successfully matched, the vectorized scan operators included in the multiple subqueries are matched. When merging the plurality of vectorized scan operators in the second execution plan into a target vectorized scan operator, the data processing method further includes: The computational operators included in the multiple subqueries are merged into the target computational operator.

13. The data processing method according to claim 12, characterized in that, The calculation operators include filtering operators, and the matching process for the calculation operators included in the multiple subqueries includes: When the filtering operators included in the multiple subqueries are at the same execution level, it is determined that the filtering operators included in the multiple subqueries are successfully matched. The step of merging the computational operators included in the multiple sub-queries into a target computational operator includes: The target filtering operator is obtained by performing a union operation on the filtering operators included in the multiple subqueries.

14. The data processing method according to claim 12, characterized in that, The computational operators include non-filtering operators, and the matching process for the computational operators included in the multiple subqueries includes: When the non-filtering operators included in the multiple subqueries are at the same execution level, have the same operator type, and target the same data column, it is determined that the non-filtering operators included in the multiple subqueries are successfully matched. The step of merging the computational operators included in the multiple sub-queries into a target computational operator includes: The non-filtered operator in any subquery is identified as the target non-filtered operator.

15. A data processing apparatus, characterized in that, include: The receiving module is used to receive data operation statements for the data storage system; The parsing module is used to parse and process the data operation statement to obtain a first execution plan. The first execution plan includes a row scan operator for a data table that uses columnar storage in the data storage system. The data table includes a metadata column and a business data column. The optimization module is used to convert the row-based scan operator in the first execution plan into a vectorized scan operator to obtain a second execution plan. The scan output column of the vectorized scan operator includes the metadata column and the target business data column. The metadata column is used to construct the scan filtering conditions of the vectorized scan operator. The execution module is used to start the vectorized scan operator in the second execution plan, read the metadata column in the metadata file corresponding to the data table to obtain metadata, and fill the metadata into the scan filter conditions; The execution module is further configured to filter the business data files in the data storage system according to the scanning filtering conditions filled with the metadata, obtain the target business data file corresponding to the data table, and perform columnar reading of the target business data columns in the target business data file to obtain the target business data.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the data processing method according to any one of claims 1 to 14.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the data processing method according to any one of claims 1 to 14.

Images