Data processing method and device, medium and electronic equipment

Abstract

The invention provides a data processing method and device, a medium and electronic equipment. In the method, after a data operation statement submitted by a user is received, a data type of to-be-processed data corresponding to the data operation statement can be determined as a target data type, and then when it is determined that the target data type is a fixed-length data type, the to-be-processed data corresponding to the data operation statement can be processed. If the target data type is determined to be the variable-length data type, continuously storing each data item contained in the to-be-processed data read from the target storage medium in the memory, and recording the null data item in the to-be-processed data through the corresponding null value bitmap; if yes, the length description array, the pointer array and the data item of the to-be-processed data read from the target storage medium are separately stored, and then after the to-be-processed data are loaded into the memory, the vectorization execution engine is called, the to-be-processed data are read from the memory, and data processing is conducted on the to-be-processed data. And obtaining an execution result of the data operation statement.

Description

Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data processing method, apparatus, medium, and electronic device. Background Technology

[0002] With the advent of the big data era, the scale of data that enterprises need to store and manage through database systems is growing exponentially, and this ever-increasing data volume is placing increasingly higher demands on the real-time analysis capabilities of database systems. Traditional row-based database engines use a row-by-row processing mode, which requires frequent function calls and branch predictions when executing complex queries (such as aggregation, filtering, and other OLAP operations), resulting in high CPU overhead and significantly increased response latency for large-scale data analysis tasks. To overcome these challenges, vectorized execution engines are being used more and more widely, such as DuckDB and Milvus. Vectorized execution engines significantly improve the execution efficiency of complex queries by adopting column-based storage and batch processing methods.

[0003] However, when using a vectorized execution engine to process data, the data processing efficiency may be reduced due to the organization format of the data in memory. This organization format consists of contiguous data description units and the actual data. A data description unit is a structure containing information such as a null value description, data length (len), and a data pointer (ptr). For example, in a memory space, a null value description (null: 0), data length (len: 2), and data pointer (ptr: 0x7e00) are stored contiguously; here, 0x7e00 is the address of the actual data.

[0004] At this point, the vectorized execution engine needs to construct or load the structure corresponding to the data (i.e., the data description unit) every time it needs to read or write data, so as to realize the reading and writing of data through the data description unit. Constructing or loading the data description unit corresponding to the data is itself an operation that requires a lot of additional overhead, resulting in low efficiency in accessing data and generating additional computational overhead. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data processing method is provided, comprising:

[0007] Statements for receiving data;

[0008] Parse the data manipulation statement to determine the data type of the data to be processed required to execute the data manipulation statement, and use it as the target data type.

[0009] When the target data type is determined to be a fixed-length data type, the data to be processed is read from the target storage medium according to the data operation statement, and the data to be processed is converted into a first data format and stored in memory; the first data format is used to store each data item contained in the data to be processed continuously in memory, and to record the empty data items in the data to be processed through the corresponding null bitmap;

[0010] When the target data type is determined to be a variable-length data type, the data to be processed is read from the target storage medium according to the data operation statement, and the data to be processed is converted into a second data format and stored in memory. The second data format is used to store the data description information and data items of the data to be processed separately. The data description information includes a length description array and a pointer array. The length description array is used to characterize the length of each data item contained in the data to be processed, and the pointer array is used to characterize the address of each data item contained in the data to be processed.

[0011] The vectorized execution engine is invoked to read the data to be processed from the memory and execute the operation corresponding to the data operation statement to process the data and obtain the execution result of the data operation statement.

[0012] According to a second aspect of one or more embodiments of this specification, a data processing apparatus is provided, comprising:

[0013] The receiving module is used to receive data operation statements;

[0014] The parsing module is used to parse the data operation statement and determine the data type of the data to be processed required to execute the data operation statement, which is then used as the target data type.

[0015] The first determining module is used to, when determining that the target data type is a fixed-length data type, read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a first data format and store it in memory; the first data format is used to store each data item contained in the data to be processed continuously in memory, and record the empty data items in the data to be processed through the corresponding null bitmap;

[0016] The second determining module is used to, when determining that the target data type is a variable-length data type, read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a second data format, and store it in memory; the second data format is used to store the data description information and data items of the data to be processed separately, the data description information includes: a length description array and a pointer array, the length description array is used to characterize the length of each data item contained in the data to be processed, and the pointer array is used to characterize the address of each data item contained in the data to be processed;

[0017] The execution module is used to call the vectorized execution engine, read the data to be processed from the memory, execute the operation corresponding to the data operation statement, process the data to be processed, and obtain the execution result of the data operation statement.

[0018] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the executable instructions to implement the steps of the data processing method described above.

[0019] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the data processing method described above.

[0020] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0021] As can be seen from the above embodiments, this specification first receives and parses data operation statements to determine the data type of the data to be processed required for executing the data operation statements, which is then used as the target data type. When the target data type is determined to be a fixed-length data type, the data to be processed is read from the target storage medium according to the data operation statements, and the data to be processed is converted into a first data format and stored in memory. The first data format is used to store each data item contained in the data to be processed consecutively in memory, and records empty data items in the data to be processed through a corresponding null bitmap. When the target data type is determined to be a variable-length data type, the data to be processed is read from the target storage medium according to the data operation statements. The system reads the data to be processed from the storage medium, converts it into a second data format, and stores it in memory. The second data format is used to separate the data description information and data items of the data to be processed. The data description information includes a length description array and a pointer array. The length description array is used to represent the length of each data item contained in the data to be processed, and the pointer array is used to represent the address of each data item contained in the data to be processed. After the data to be processed is loaded into memory, the vectorized execution engine can be invoked to read the data to be processed from memory and execute the operations corresponding to the data operation statements to process the data and obtain the execution results of the data operation statements.

[0022] This method addresses the overhead of constructing and accessing data description units (i.e., structures composed of null values, lengths, pointers, etc.) by employing differentiated memory organization formats for different data types. For fixed-length data types, storing data items contiguously supplemented with independent null value bitmaps achieves a compact data layout and efficient vectorized access, avoiding the overhead of constructing description structures one by one. For variable-length data types, the length value and address pointer of each data item are stored as length arrays and pointer arrays, respectively. This allows the vectorized execution engine to access length and address information in batches during processing, significantly reducing the read / write overhead caused by frequent parsing of complex structures. Furthermore, the database system can complete format conversion during the data loading stage, enabling subsequent vectorized calculations to be performed directly based on the optimized memory layout. This significantly improves data reading efficiency and computational performance, effectively reduces query latency, and enhances the real-time analysis capabilities of the database system. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of a database system provided in an exemplary embodiment.

[0024] Figure 2 This is a schematic diagram of a storage format provided in an exemplary embodiment.

[0025] Figure 3 This is a schematic flowchart of a data processing method provided in an exemplary embodiment.

[0026] Figure 4 This is a schematic diagram of a first data format provided in an exemplary embodiment.

[0027] Figure 5 This is a schematic diagram of a second data format provided in an exemplary embodiment.

[0028] Figure 6 This is a schematic diagram of a third data format provided in an exemplary embodiment.

[0029] Figure 7 This is a schematic diagram of a sorting key provided in an exemplary embodiment.

[0030] Figure 8 This is a schematic structural diagram of a device provided in an exemplary embodiment.

[0031] Figure 9 This is a block diagram of a data processing apparatus provided in an exemplary embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0033] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0034] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram of the overall architecture of a database system provided in an exemplary embodiment.

[0036] like Figure 1 As shown, a database system can contain three components, and the functions of each component are as follows:

[0037] Parser: After receiving the data operation statement sent by the user, the parser can use the lexical / syntax parsing module to divide the received data operation statement into individual words and parse the entire data operation statement according to the pre-defined grammar rules. Thus, the parser can convert the data operation statement into an in-memory data structure with grammatical structure information, i.e., a syntax tree.

[0038] After the syntax tree is generated, the semantic parsing module can further convert it into an internal data structure with database semantic information. In this process, the semantic parsing module translates each field (token) in the SQL request into a corresponding object (e.g., database, table, column, index, etc.) based on the database metadata, and generates a statement tree.

[0039] To facilitate the generation of the optimal execution plan, a logic rewriting module can be used after the semantic parsing module to further analyze the semantics of the data operation statements. Based on preset rewriting rules or cost models, an equivalent transformation is performed on the original statement tree to rewrite the received data operation statements into other equivalent forms, which are then provided to the subsequent optimizer for further optimization. The result of the equivalent transformation performed on the original statement tree by the logic rewriting module is still a statement tree.

[0040] The optimizer comprehensively considers various factors such as the semantics of the data manipulation statement, the data characteristics of the objects involved in the statement, and the physical distribution of the objects. It performs operations such as access path selection, join order selection, join algorithm selection, and distributed plan generation to ultimately generate an optimal execution plan corresponding to the data manipulation statement. The execution plan of a data manipulation statement is an execution tree composed of multiple operators. If the optimizer determines that the operation corresponding to the data manipulation statement is a transaction processing (TP) operation, it can call the row-store cost model to evaluate the execution cost based on row-store operations, including I / O and CPU overhead for data reading, updating, inserting, and deleting. Based on the output of the row-store cost model, it generates the execution plan. Conversely, if the optimizer determines that the operation corresponding to the data manipulation statement is an Analytical Processing (AP) operation, it can invoke the Column-store Cost Model to evaluate the execution cost based on column-store, including: batch reading of column data, vectorized computation, compression efficiency, and the cost of reading only relevant columns (column pruning). Based on the output of the Column-store Cost Model, an execution plan is generated. Furthermore, a code generator can convert the optimal execution plan generated by the optimizer into executable code.

[0041] Executor: Initiates the execution process of data manipulation statements, starting with the operators at the top of the execution plan. The entire execution process is completed by the logic of the operators themselves, and the execution result is returned. Before the execution phase begins, the executor determines whether to use a vectorized or non-vectorized execution engine based on the execution plan generated by the optimizer. (When generating the execution plan, the optimizer can determine whether to generate an execution plan using vectorized or non-vectorized operators based on the type of data operation statement (e.g., TP / AP), the data storage format on disk (e.g., row storage / column storage), statistics, resource conditions, etc.) Taking the vectorized execution engine as an example, in the data access phase, scan operators (e.g., Vectorized Table Scan, Vectorized Index Scan, Vectorized Index Seek, etc.) are called to read the required data to be processed from the storage engine and store it in memory according to the format of data description units and data items being stored separately. Then, data processing operators (e.g., Filter, Project, Join, Sort, Aggregate, Group By, etc.) can be called to process the data to be processed, obtain the execution results of the data operation statements, and return them.

[0042] When using a vectorized execution engine to process data, the data to be processed is organized in memory into contiguous data description units and actual data, such as... Figure 2 As shown.

[0043] Figure 2 This is a schematic diagram of a storage format provided in an exemplary embodiment.

[0044] Combination Figure 2 It can be seen that currently, the data to be processed in memory is stored in a format where contiguous data description units (i.e., structures composed of information such as null, data length (len), and data pointer (ptr)) and the actual data are stored separately. This means that accessing any data requires first loading the data description unit, then reading the pointer from the data description unit, and finally accessing the data through the pointer. This double indirect access (descriptor->ptr->data) increases latency and instruction overhead, which is the root cause of low read / write efficiency. Furthermore, because this data storage format requires maintaining a structure (i.e., data description unit) to manage the data, and this structure itself occupies additional memory, the memory space occupied by the data to be processed increases. Additionally, although in Figure 2Logically, data items are stored as a continuous column of data. However, in actual storage space, the data to be processed is not necessarily stored continuously. This makes it impossible to use Single Instruction Multiple Data (SIMD) technology to batch process the data when processing it through a vectorized execution engine, which in turn reduces the execution efficiency of data manipulation statements.

[0045] Based on this, this specification provides a data processing method. The technical solutions provided by the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0046] Figure 3 This is a flowchart illustrating a data processing method provided in an exemplary embodiment, including:

[0047] S300: Data receiving operation statement.

[0048] S302: Parse the data operation statement to determine the data type of the data to be processed required to execute the data operation statement, and use it as the target data type.

[0049] In this specification, the entity executing the data processing method can refer to a database system or an executor within that database system. From a hardware perspective, database systems typically run on specific types of computing devices. Therefore, the entity executing the data processing method can also refer to the physical or virtual computing device on which the database system is deployed, such as a server, desktop computer, or laptop computer. In a distributed database architecture, the entity executing the data processing method can also refer to the computing device that performs core coordination and management functions, such as a coordination server, management server, or independent coordinator; this specification does not impose any limitations on this.

[0050] For ease of description, this manual treats the software-level database system and its dependent hardware resources as a single logical unit, collectively referred to as the database system. The following text will use this database system as the execution entity to illustrate the data processing methods provided in this manual.

[0051] Specifically, a database system can receive data manipulation statements submitted by users and perform operations such as data storage, data management, and data retrieval based on the received data manipulation statements.

[0052] The aforementioned data manipulation statements can refer to programmatic instructions written by users using a specific database query language, used to explicitly instruct the database system to perform corresponding data processing operations. These database query languages ​​can include SQL (Structured Query Language), XQuery, MongoDB MQL, etc. Taking SQL as an example, the data manipulation statements received by the database system can include the following categories: INSERT statements for inserting new data into a data table, UPDATE statements for modifying existing data in a data table, DELETE statements for deleting existing data from a data table, and SELECT statements for retrieving existing data from a data table, etc.

[0053] During this process, after receiving the data operation statement submitted by the user, the database system can parse the data operation statement to determine the data type of the data to be processed required to execute the data operation statement, which is then used as the target data type.

[0054] In practical applications, the data to be processed required to execute data manipulation statements may be the values ​​of multiple fields in a data table. In this case, the database system can treat the value of each of the above fields as the data to be processed, and then determine the data type of the data to be processed for each field, and use it as the target data type of the data to be processed.

[0055] For example: If the data manipulation statement submitted by the user is:

[0056] SELECT id, name, salary * 1.1

[0057] FROM employees

[0058] WHERE hire_date>'2022-01-01'

[0059] That is, from the data table named employees, query and return the employee ID and name of employees whose hiring date (hire_date) is after January 1, 2022, and determine the new salary after increasing the original salary of these employees by 10%.

[0060] In the above data manipulation statement, the data to be processed required to execute the data manipulation statement can include four fields: the first field is the employee's employee number (id), the second field is the employee's name (name), the third field is the employee's original salary (salary), and the fourth field is the employee's hire date (hire_date).

[0061] In this data structure, the employee's ID (id) is of data type int, the employee's starting salary (salary) is of data type Declmal, and the employee's hire date (hire_date) is of data type Date. Since int, Declmal, and Date are all fixed-length data types, the target data type for the values ​​of these three fields is a fixed-length data type. The employee's name (name) is of data type Varchar, which is a variable-length character data type. Therefore, the target data type for the values ​​of this field is a variable-length data type.

[0062] Based on this, the database system can retrieve the values ​​of four fields—employee ID (id), employee name (name), employee's original salary (salary), and employee's hire date (hire_date)—from the employee table stored on disk via the storage engine and store them in memory. The vectorized execution engine then retrieves these four field values ​​from memory and uses SIMD instructions to perform batch comparisons of the hire date (hire_date) values ​​to filter out employees whose hire date is later than '2022-01-01'. Next, it performs a vectorized multiplication operation (*1.1) on the employee's original salary (salary) value. Finally, it projects only the employee ID (id) and employee name (name) of the rows that meet the conditions, thus obtaining the execution result of the above data manipulation statement and returning it.

[0063] S304: When the target data type is determined to be a fixed-length data type, the data to be processed is read from the target storage medium according to the data operation statement, and the data to be processed is converted into a first data format and stored in memory; the first data format is used to store each data item contained in the data to be processed continuously in memory, and to record the empty data items in the data to be processed through the corresponding null bitmap.

[0064] S306: When the target data type is determined to be a variable-length data type, the data to be processed is read from the target storage medium according to the data operation statement, and the data to be processed is converted into a second data format and stored in memory; the second data format is used to store the data description information and data items of the data to be processed separately, the data description information includes: a length description array and a pointer array, the length description array is used to characterize the length of each data item contained in the data to be processed, and the pointer array is used to characterize the address of each data item contained in the data to be processed.

[0065] Furthermore, when the database system determines that the target data type of the data to be processed is a fixed-length data type, it can read the data to be processed according to the physical layout of the data to be processed in the target storage medium (such as row storage, column storage, etc.) according to the data operation statement, and then convert the data to be processed into the first data format and store it in memory. This can be understood as reorganizing the data to be processed into the first data format and storing it in memory.

[0066] The target storage medium mentioned above can refer to a physical or logical device used for persistent storage of data, such as a disk or a distributed file system.

[0067] The first data format described above is used to store each data item in the data to be processed consecutively in memory, and to record empty data items in the data to be processed through a corresponding null bitmap, as follows: Figure 4 As shown.

[0068] Figure 4 This is a schematic diagram of a first data format provided in an exemplary embodiment.

[0069] exist Figure 4 In this database, the data to be processed, stored in the first data format, contains three data items: the first data item is 10, the second is null, and the third is 100. Each data item occupies the same storage space and is stored contiguously in memory. Because each data item occupies the same storage space, the database system does not need to record the length information of each data item separately; it only needs to use a separate null bitmap (0, 1, 0) to record which data items are null.

[0070] As can be seen from the above, the database system can convert the data to be processed into a first data format, so that the vectorized execution engine can directly locate any data item through a simple address offset without pointer jumps or multiple memory dereference operations, thereby improving the cache hit rate and access efficiency of the data to be processed.

[0071] In addition, since each data item in the first data format is stored in a contiguous manner, the vectorized execution engine can load multiple contiguous data items at once and perform parallel processing (such as batch comparison, arithmetic operations, etc.) through the SIMD instruction set, thereby significantly improving the execution efficiency of typical analysis operations such as aggregation, filtering, and expression calculation.

[0072] When a database system determines that the target data type of the data to be processed is a variable-length data type, it can read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a second data format, and then store it in memory.

[0073] In the above content, the second data format is used to separately store the data description information and data items of the data to be processed. The data description information includes: a length description array and a pointer array. The length description array represents the length of each data item in the data to be processed, and the pointer array represents the address of each data item in the data to be processed, specifically as follows: Figure 5 As shown.

[0074] Figure 5 This is a schematic diagram of a second data format provided in an exemplary embodiment.

[0075] from Figure 5 As can be seen, using the second data format allows for the separate storage of the data description information and data items of the data to be processed. The data description information can be organized into a length description array and a pointer array. Figure 5 In the above-mentioned length description array, there can be three values: 2, 0, and 4, which respectively represent the length of the first data item "ab" as 2, the length of the second null data item as 0, and the length of the third data item "2533" as 4. Figure 5 In the above pointer array, there may be three address pointers: 0x7e00, null, and 0x7e08, which are used to represent the actual storage location (i.e., address) of the above three data items in memory.

[0076] When a database system converts data to a second data format and stores it in memory, the actual data items may not be stored contiguously in the same memory area, but rather distributed according to memory usage. In this case, each data item is located and accessed through its corresponding length value and address pointer in both the length array and the pointer array, thus enabling the database system to efficiently access non-contiguously stored data.

[0077] As can be seen from the above, when a database system needs to access data to be processed that is stored in memory and organized according to the second data format, it can directly obtain the length value and address pointer corresponding to each data item from the length array and the pointer array, and access the data item based on the obtained length value and address pointer, without having to first construct or load the data description unit and parse the data description unit to obtain the address pointer corresponding to the data item, and finally access the data item through the address pointer corresponding to the data item. This can significantly improve the efficiency of the database system in accessing non-contiguous data.

[0078] In addition, under normal circumstances, when performing projection operations, the vectorized execution engine needs to copy each data item and its data description unit into a new continuous buffer to reorganize the data items and their data description units into a continuous format. This satisfies the requirements of the vectorized projection operator for data continuity and alignment, and allows the vectorized projection operator to be run to obtain the projection result.

[0079] However, when the data to be processed is converted to the second data format described above and stored in memory, since the address pointer and length value of each data item have been organized and centrally managed as an array, the vectorized execution engine can directly reuse the existing length array and pointer array to construct logical references for each data item during the projection operation, without performing physical copying. In subsequent projection operations, only the length array and pointer array of these data items need to be passed, and combined with the values ​​of each data item, on-demand access or lazy materialization can be performed at runtime to complete the projection operation on the data to be processed and obtain the projection result.

[0080] Furthermore, because the length array and pointer array mentioned above can be accessed and computed quickly independently of the data items, when the vectorized execution engine performs operations such as conditional filtering, predicate pushdown, or short-circuit computation, it can first predict and filter the data items based on the length array and pointer array. This allows the database system to avoid fully decoding, copying, or organizing the entire data to be processed, and only to process the truly relevant data items, thus significantly improving the execution efficiency of these operations.

[0081] For example, by checking the length array, you can quickly identify null values ​​(e.g., if the length of a data item is 0, then the data item is null) or very short strings, or combine predicate conditions to determine whether it is possible to meet the query conditions without decoding the complete data. For data items that obviously do not meet the conditions, you can directly skip the access process of their corresponding data items, thereby avoiding accessing all data items of the data to be processed.

[0082] In practical applications, to further improve the efficiency of the vectorized execution engine in performing data processing operations on data stored in the second data format described above, the data description information may also include: an array of null values, such as... Figure 5 The empty value array shown contains three values: 0, 1, and 0, which respectively represent that the first data item ab is not empty, the second data item is empty, and the third data item 2533 is not empty.

[0083] Furthermore, when the database system determines that the target data type is neither a fixed-length data type nor a variable-length data type, that is, when the database system determines that the target data type is another data type, it can read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a third data format, and then store it in memory.

[0084] The third data format is used to store variable-length data contiguously in memory, and records the starting position of each data item through an offset array, specifically as follows: Figure 6 As shown.

[0085] Figure 6 This is a schematic diagram of a third data format provided in an exemplary embodiment.

[0086] Combination Figure 6 It can be seen that the data to be processed, stored in the third data format, contains three data items: the first data item is "ab", the second is an empty value, and the third is "2533". The first and third data items are stored contiguously in memory, while the second data item does not occupy storage space. Furthermore, the length and offset of each data item are described by an offset array, such as... Figure 5 In the sequence 0, 2, 2, 6, the starting offset is 0, the offset after the first data item is 2, the offset after the second data item is still 2, and the offset after the third data item is 6.

[0087] In addition, for the data to be processed stored in the third data format, the database system can also record which data items are empty through the null bitmap 0,1,0.

[0088] As can be seen from the above, when the database system reads the data to be processed from the disk into memory, it can select a suitable format from the three data formats mentioned above based on the target data type of the data to be processed required for executing data operation statements. This dynamic selection and conversion mechanism can significantly improve data processing efficiency and optimize the execution performance of data operation statements. The following describes in detail the process by which the database system processes the data to be processed after reading it into memory, as shown in step S308.

[0089] S308: Invoke the vectorized execution engine, read the data to be processed from the memory, and execute the operation corresponding to the data operation statement to process the data to be processed and obtain the execution result of the data operation statement.

[0090] In this specification, after the database system reads the data to be processed into memory, it can invoke the vectorized execution engine to read at least a portion of the data to be processed from memory as a batch dataset. Subsequently, the database system can determine the attribute information corresponding to this batch dataset and optimize the execution strategy and operation scheduling based on the determined attribute information. This allows for efficient execution of the operations corresponding to the data operation statements to process the batch dataset and obtain the execution results of the data operation statements.

[0091] The aforementioned attribute information includes at least one of the following: null value distribution state, which data items in the batch dataset are null values; data continuity state, which indicates whether the batch dataset is stored in a continuous manner; and row filtering state, which indicates whether each data item in the batch dataset has been filtered.

[0092] The aforementioned null value distribution state can be determined by the vectorized execution engine when reading the data to be processed, based on the null value bitmap or null value array of the data to be processed, and recorded in the metadata of the batch dataset.

[0093] The aforementioned data continuity state can be determined by the vectorized execution engine based on the metadata of the data to be processed when reading it, and recorded in the metadata of the batch dataset. The metadata of the data to be processed is created and maintained when the data is read into memory.

[0094] The row filtering state described above can be generated by the vectorized execution engine before performing filtering or conditional judgment operations, based on the null bitmap, offset array, null array, pointer array, length array, etc., of the data to be processed. In practical applications, this row filtering state can be represented by a selection vector or a mask to indicate which rows in the current batch dataset might meet the conditions (and need to be retained), and which rows clearly do not meet the conditions (and need to be filtered out). Of course, this row filtering state can also be represented using a Boolean array or other similar methods.

[0095] As can be seen from the above, during the execution process, the vectorized execution engine can dynamically maintain the attribute information of the batch dataset. This attribute information is directly used to execute the operations corresponding to the data operation statements, thereby improving the execution efficiency of the data operation statements during the data processing process.

[0096] Specifically, a database system can execute data manipulation statements in multiple ways based on the attribute information of a batch dataset.

[0097] For example, if the database system determines that there are no null values ​​in the batch dataset based on the null value distribution, then there is no need to perform calculation operations to handle null values ​​during the data processing of the batch dataset. This reduces redundant condition judgment operations required during the data processing of the batch dataset, thereby improving the execution efficiency of data operation statements.

[0098] For example, if the database system determines that the data continuity status of the batch dataset is continuous and the row filtering status is unfiltered based on the attribute information, then the SIMD instruction can be directly called to load the data to be processed in the batch dataset in batches, so as to perform data processing on the data to be processed in the batch dataset in parallel, thereby improving the execution efficiency of data operation statements.

[0099] In addition, in practical applications, some data processing statements often require sorting of the data to be processed before execution. To improve the efficiency of sorting operations on the data to be processed, the database system can also call a vectorized execution engine to read the data to be processed from memory. If the operation corresponding to the data processing statement is a sorting operation, the system determines the sort key used for the sorting operation. This allows for the materialization of at least a portion of the data to be processed that serves as the sort key, resulting in a materialized data sequence. Sorting this materialized data sequence yields a sorted data sequence. Specifically, as follows... Figure 7 As shown.

[0100] Figure 7This is a schematic diagram of a sorting key provided in an exemplary embodiment.

[0101] Combination Figure 7 As can be seen, the employee table can contain four fields: employee ID (id), employee name (name), employee's initial salary (salary), and employee's hire date (hire_date). In this case, the data manipulation statement would be:

[0102] SELECT*FROM employees ORDER BY hire_date ASC

[0103] In other words, the employee information is sorted in ascending order according to their hire date and then output. In this data manipulation statement, the hire date (hire_date) field used for sorting is the sort key, and the hire date values ​​for different employees under this field constitute at least a portion of the data to be processed, which serves as the sort key. At this point, the database system can use a vectorized execution engine to extract this portion of the data to be processed, which serves as the sort key, and materialize it into a "vector." Each element of this vector represents a value of the sort key, so that subsequent sorting operations only need to read and compare the element values ​​in this vector, reducing the amount of data accessed and the number of accesses, thereby improving the execution efficiency of the data manipulation statement.

[0104] In addition, when there are multiple sort keys, the database system can determine the data corresponding to each sort key in the data to be processed, as the original sort value. Then, for each data row in the table composed of the original sort values ​​of each sort key, the original sort values ​​contained in that data row can be mapped to the corresponding encoded value. The encoded value sequence is then materialized to obtain a materialized data sequence. Finally, the materialized data sequence can be sorted to obtain a sorted data sequence.

[0105] Specifically, for any two data rows, the sorting result of sorting the two data rows according to the size of their corresponding encoded values ​​is consistent with the sorting result of sorting the two data rows according to their original sorting values.

[0106] For example: when the data manipulation statement is:

[0107] SELECT * FROM employees

[0108] ORDER BY hire_date ASC,salary ASC

[0109] That is, after sorting and outputting the employee information according to the employee's start date and salary, the database system will first sort the information of each employee from the earliest to the latest (i.e., ascending order ASC) according to the start date. For employees with the same start date, the information of these employees can be sorted from the lowest to the highest salary (i.e., ascending order ASC) according to their salary.

[0110] As can be seen from the above, the data manipulation statement uses two sort keys: the employee's hire date (hire_date) and the employee's base salary (salary). Each row in the table, composed of the base sort values, contains two values: each employee's hire date and base salary. To improve the efficiency of the data manipulation statement, the database system can map the hire date and base salary values ​​of each row to a single, comparable coded value. Furthermore, sorting any two rows according to their coded values ​​will produce the same result as sorting by the base sort values. This allows for sorting based on the coded values ​​of each row, reducing the amount of data accessed and the number of accesses, thus improving the efficiency of the data manipulation statement.

[0111] As mentioned above, there are several ways to map the employee's start date and original salary value that make up the data row to a single, comparable coded value.

[0112] For example: convert the values ​​of multiple sort keys to strings, and concatenate the values ​​of multiple sort keys into a single string (i.e., the encoded value), where the strings corresponding to each sort key value are connected using a delimiter that does not appear in the original data. For example: Figure 7 The string obtained by concatenating the start date and initial salary of an employee named Alice is 2023-01-15 + \0 + 7000 = 2023-01-15\07000. Similarly, the string obtained by concatenating the start date and initial salary of an employee named Bob is 2022-03-10\06000, and so on. Based on this, the database system can directly perform a lexicographical comparison of the encoded values ​​of these data rows, where 2022-03-10\06000 < 2023-01-15\07000, and sort these data rows according to the comparison result.

[0113] For another example: assign a weight to each value of the sorting key, so as to perform a weighted sum of each value of the sorting key to construct a numerical value (i.e., the encoded value). For example, Figure 7 set the weight of the sorting key of the employee's hire date in Figure 7 to 100,000, and set the weight of the sorting key of the employee's original salary to 1. At this time, Figure 7 the encoded value obtained by weighted summing the hire date and the original salary values of the employee named Alice in Figure 7 is 20230115 * 100,000 + 7000 * 1 = 2023011507000. Similarly, the encoded value obtained by weighted summing the hire date and the original salary values of the employee named Bob is 20220310 * 100,000 + 6000 * 1 = 2022031006000, and so on. Based on this, the database system can directly compare the encoded values of these data rows, 2022031006000 <2023011>507000, and sort these data rows according to the comparison results.

[0114] In addition, in order to improve the efficiency of performing projection operations through the vectorized execution engine, the database system can also call the vectorized execution engine to read the data to be processed from the memory. When it detects that the operation type corresponding to the current data operation statement is a projection operation, the database system can obtain the metadata of the data to be processed. The metadata of the data to be processed includes at least one of the data type of the data to be processed, the column length of the data to be processed, and the null value information of the data to be processed.

[0115] Based on the obtained metadata of the data to be processed, filter from the pre-constructed candidate code library. The candidate code library stores candidate codes for projection calculations optimized for different metadata combinations (for example: for columns with a fixed-length data type (such as INT) and no null values, set unconditional judgment vectorized data copying or arithmetic operation codes. For columns with a variable-length data type (such as VARCHAR) and null values, set codes that include efficient null bitmap processing logic and vectorized string operations (such as truncation, concatenation), etc.). Then, according to the metadata of the data to be processed, select a candidate code that matches the metadata of the data to be processed from each candidate code as the target code. Then, the target code can be run to perform projection calculations on the column data of the data table composed of the data to be processed to obtain the projection result.

[0116] Furthermore, after the database system executes the operations corresponding to the data operation statements through the vectorized execution engine and processes the data to be processed to obtain the execution results of the data operation statements, it can also materialize the execution results, that is, store the execution results in a temporary table on the disk. This allows the system to directly retrieve and use the execution results of the current data operation statement when responding to subsequent data operation statements that are identical to the current one, thereby improving the execution efficiency of subsequent data operation statements.

[0117] When materializing the execution results of the above data manipulation statements, the third data format can be used for storage. Since the length and offset position of each data item in the third data format are described by an offset array, the efficiency of traversing a large number of data items in a short time can be improved while avoiding the waste of storage resources caused by each data item occupying a fixed length.

[0118] As can be seen from the above, database systems can solve the problem of additional construction and access overhead caused by the contiguous storage of data description units (i.e., structures composed of null values, lengths, pointers, etc.) by adopting differentiated memory organization formats for different data types. Specifically, for fixed-length data types, storing data items contiguously supplemented with independent null value bitmaps achieves a compact data layout and efficient vectorized access, avoiding the overhead of constructing description structures one by one. For variable-length data types, the length value and address pointer of each data item are stored as length arrays and pointer arrays respectively, allowing the vectorized execution engine to access length and address information in batches during processing, significantly reducing the read / write overhead caused by frequent parsing of complex structures. Furthermore, the database system can complete format conversion during the data loading stage, allowing subsequent vectorized calculations to be performed directly based on the optimized memory layout, thereby greatly improving data reading efficiency and computational performance, effectively reducing query latency, and enhancing the real-time analysis capabilities of the database system.

[0119] Figure 8 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 8As shown, device 800 mainly consists of a communication interface 802, a user interface 804, a processor 806, and a data storage 808. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 810. The communication interface 802 enables device 800 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 802 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 802 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 802 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 802 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0120] User interface 804 includes receiving user input and providing output to the user. Therefore, user interface 804 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 804 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 804 may include software, circuitry, or other forms of logic capable of transmitting and receiving data to and from external user input / output devices. Additionally or alternatively, device 800 may support remote access from other devices via communication interface 802 or another physical interface (not shown). User interface 804 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 804 may also be configured as a display device for rendering or displaying text fragments.

[0121] The processor 806 may contain one or more general-purpose processors and / or special-purpose processors.

[0122] Data storage 808 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 806. Data storage 808 may include removable and non-removable components.

[0123] Processor 806 is capable of executing program instructions 818 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 808 to perform the various functions described herein. Data storage 808 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 800, enable device 800 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 818 by processor 806 may result in processor 806 using data 812.

[0124] For example, program instructions 818 may include an operating system 822 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 800 and one or more application programs 820 (e.g., a browser, social application, or game application). Similarly, data 812 may include operating system data 816 and application data 814. Operating system data 816 is primarily accessible to the operating system 822, while application data 814 is primarily accessible to one or more application programs 820. Application data 814 may reside in a file system visible or hidden from the user of device 800.

[0125] Application 820 can communicate with operating system 822 through one or more application programming interfaces (APIs). These APIs help application 820 read and / or write application data 814, transmit or receive information via communication interface 802, receive or display information on user interface 804, etc.

[0126] In some terminology, application 820 may be simply referred to as "app". Furthermore, application 820 can be downloaded to device 800 through one or more online app stores or app markets. However, applications can also be installed on device 800 in other ways, such as through a web browser or a physical interface on device 800 (e.g., a USB port).

[0127] Please refer to Figure 9 Data processing devices can be applied to, for example Figure 8 The device shown is used to implement the technical solution of this specification. The data processing apparatus may include:

[0128] Receiver module 901 is used to receive data operation statements;

[0129] Parsing module 902 is used to parse the data operation statement and determine the data type of the data to be processed required to execute the data operation statement, as the target data type;

[0130] The first determining module 903 is used to, when determining that the target data type is a fixed-length data type, read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a first data format and store it in memory; the first data format is used to store each data item contained in the data to be processed continuously in memory, and record the empty data items in the data to be processed through the corresponding null bitmap;

[0131] The second determining module 904 is used to, when determining that the target data type is a variable-length data type, read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a second data format, and store it in memory; the second data format is used to store the data description information and data items of the data to be processed separately, the data description information includes: a length description array and a pointer array, the length description array is used to characterize the length of each data item contained in the data to be processed, and the pointer array is used to characterize the address of each data item contained in the data to be processed;

[0132] The execution module 905 is used to call the vectorized execution engine, read the data to be processed from the memory, and execute the operation corresponding to the data operation statement to process the data to be processed and obtain the execution result of the data operation statement.

[0133] Optionally, the execution module 905 is specifically configured to: invoke the vectorized execution engine to read at least a portion of the data to be processed from the memory as a batch dataset; determine the attribute information corresponding to the batch dataset, the attribute information including at least one of: null value distribution state, data continuity state, and row filtering state; and execute the operation corresponding to the data operation statement according to the attribute information to process the batch dataset.

[0134] Optionally, the execution module 905 is specifically configured to, when the data continuity status is determined to be continuous based on the attribute information and the row filtering status is unfiltered, call SIMD instructions to batch load the data to be processed in the batch dataset; and perform parallel data processing on the data to be processed in the batch dataset.

[0135] Optionally, the execution module 905 is specifically configured to: invoke the vectorized execution engine, read the data to be processed from the memory, and, if the operation corresponding to the data operation statement is a sorting operation, determine the sorting key used to perform the sorting operation; perform materialization processing on at least a portion of the data to be processed that is used as the sorting key to obtain a materialized data sequence; and sort the materialized data sequence to obtain a sorted data sequence.

[0136] Optionally, the execution module 905 is specifically configured to: when there are multiple sorting keys, determine the data corresponding to each sorting key in the data to be processed as the original sorting value; for each data row in the table composed of the original sorting values ​​of each sorting key, map each original sorting value contained in the data row to the corresponding encoded value of the data row; wherein, for any two data rows, the sorting result of sorting the two data rows according to the size of the encoded values ​​corresponding to the two data rows is consistent with the sorting result of sorting according to the original sorting values ​​contained in the two data rows; and perform materialization processing on the encoded value sequence composed of each encoded value to obtain the materialized data sequence.

[0137] Optionally, the execution module 905 is specifically configured to: invoke the vectorized execution engine to read the data to be processed from the memory; when the operation corresponding to the data operation statement is a projection operation, select candidate code from each candidate code that matches the metadata of the data to be processed as the target code; the metadata includes at least one of the data type of the data to be processed, the column length of the data to be processed, and the null value information of the data to be processed; and run the target code to perform projection calculation on each column of the data table composed of the data to be processed to obtain the projection result.

[0138] Optionally, the execution module 905 is specifically used to, when determining that the target data type is another data type, read the data to be processed from the target storage medium according to the data operation statement, convert the data to be processed into a third data format and store it in memory; the third data format is used to store variable-length data continuously in memory and record the starting position of each data item through an offset array.

[0139] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0140] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0141] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0142] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0143] What those skilled in the art will understand is:

[0144] In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0145] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0146] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0147] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0148] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0149] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0150] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A data processing method, comprising: receiving a data operation statement; parsing the data operation statement to determine a data type of data to be processed required for executing the data operation statement as a target data type; when the target data type is determined as a fixed-length data type, reading data to be processed from a target storage medium according to the data operation statement, and storing the data to be processed in a memory after converting the data to be processed into a first data format; the first data format is used to store each data item contained in the data to be processed continuously in the memory, and record data items that are empty in the data to be processed through a corresponding null bitmap; when the target data type is determined as a variable-length data type, reading data to be processed from a target storage medium according to the data operation statement, and storing the data to be processed in a memory after converting the data to be processed into a second data format; the second data format is used to store data description information and data items of the data to be processed separately, the data description information includes: a length description array used to represent the length of each data item contained in the data to be processed, and a pointer array used to represent the address of each data item contained in the data to be processed; calling a vector execution engine to read the data to be processed from the memory, and execute an operation corresponding to the data operation statement to perform data processing on the data to be processed, and obtain an execution result of the data operation statement. 2.The method of claim 1, wherein calling the vector execution engine to read the data to be processed from the memory, and execute an operation corresponding to the data operation statement to perform data processing on the data to be processed, specifically comprises: calling the vector execution engine to read at least part of the data to be processed from the memory as a batch data set; determining attribute information corresponding to the batch data set, the attribute information including at least one of: a null value distribution state, a data continuity state, and a row filtering state; performing an operation corresponding to the data operation statement on the batch data set according to the attribute information. 3.The method of claim 2, wherein performing an operation corresponding to the data operation statement on the batch data set according to the attribute information, specifically comprises: when the data continuity state is determined as continuous and the row filtering state is determined as not filtered according to the attribute information, calling a SIMD instruction to load data to be processed in the batch data set in batches; performing data processing on the data to be processed in the batch data set in parallel. 4.The method of claim 1, wherein calling the vector execution engine to read the data to be processed from the memory, and execute an operation corresponding to the data operation statement to perform data processing on the data to be processed, specifically comprises: calling the vector execution engine to read the data to be processed from the memory, and determining a sorting key used for executing a sorting operation in a case that the operation corresponding to the data operation statement is the sorting operation. materialize at least part of the to-be-processed data used as the sorting key to obtain a materialized data sequence; sort the materialized data sequence to obtain a sorted data sequence. 5.The method of claim 4, wherein the materializing at least part of the to-be-processed data used as the sorting key to obtain a materialized data sequence comprises: determining, when the sorting key is multiple, each sorting key corresponding data in the to-be-processed data as an original sorting value; mapping each original sorting value contained in each data row in a table composed of the original sorting values of each sorting key to an encoding value corresponding to the data row; wherein, for any two data rows, the sorting result of sorting the any two data rows according to the encoding values corresponding to the any two data rows is consistent with the sorting result of sorting the any two data rows according to each original sorting value contained in the any two data rows; and materializing a sequence of encoding values composed of the encoding values to obtain a materialized data sequence. 6.The method of claim 1, wherein the vectorized execution engine is invoked to read the to-be-processed data from the memory, execute the operation corresponding to the data operation statement, and perform data processing on the to-be-processed data, and the method comprises: invoking the vectorized execution engine to read the to-be-processed data from the memory; when the operation corresponding to the data operation statement is a projection operation, selecting a candidate code matching the metadata of the to-be-processed data from the candidate codes as a target code; the metadata comprises at least one of a data type of the to-be-processed data, a column length of the to-be-processed data, and null information of the to-be-processed data; and running the target code to perform projection calculation on each column data of a data table composed of the to-be-processed data to obtain a projection result. 7.The method of claim 1, further comprising: when the target data type is determined to be other data types, reading to-be-processed data from a target storage medium according to the data operation statement, and storing the to-be-processed data in the memory after converting the to-be-processed data into a third data format; the third data format is used to store variable-length data continuously in the memory and record the starting positions of each data item through an offset array. 8.A data processing apparatus, comprising: a receiving module configured to receive a data operation statement; a parsing module configured to parse the data operation statement to determine a data type of to-be-processed data required for executing the data operation statement as a target data type; and a first determining module configured to, when the target data type is determined to be a fixed-length data type, read to-be-processed data from a target storage medium according to the data operation statement, and store the to-be-processed data in the memory after converting the to-be-processed data into a first data format; the first data format is used to store each data item contained in the to-be-processed data continuously in the memory and record data items that are null in the to-be-processed data through a corresponding null bitmap. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The second determining module is configured to, when determining that the target data type is a variable-length data type, read the to-be-processed data from the target storage medium according to the data operation statement, and store the to-be-processed data in the memory after converting the to-be-processed data into a second data format. The second data format is used to store data description information and data items of the to-be-processed data separately, and the data description information includes a length description array and a pointer array. The length description array is used to represent the length of each data item contained in the to-be-processed data, and the pointer array is used to represent the address of each data item contained in the to-be-processed data. The executing module is configured to call a vectorization execution engine, read the to-be-processed data from the memory, and perform an operation corresponding to the data operation statement to process the to-be-processed data and obtain an execution result of the data operation statement.

9. An electronic device comprising: A processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method of any one of claims 1-7 by running the executable instructions.

10. A computer readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method of any one of claims 1-7.

11. A computer program product comprising computer program / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1-7.

Images