Data processing method and apparatus, database management system, electronic device, and storage medium

Abstract

One or more embodiments of the present description provide a data processing method and apparatus, an electronic device, and a storage medium. The method comprises: in response to an intermediate result generated in the process of executing a SQL statement by an operator, controlling the operator to call a temporary storage module by means of a corresponding format interface, so as to store, by means of the temporary storage module, the intermediate result on the basis of a format corresponding to the operator, wherein the temporary storage module is configured with a plurality of format interfaces, and different format interfaces are used for enabling operators that call the temporary storage module to store data on the basis of different formats. That is to say, in the method, the operators can call the temporary storage module by means of different format interfaces to customize different data formats, so that the intermediate result can be stored in the formats corresponding to the operators, meeting the format requirements of different operators, improving the storage efficiency of the intermediate result.

Description

Data processing method and device, database management system, electronic device and storage medium Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to data processing. Background Art

[0002] With the rapid development of the internet and information technology, data generation is exploding, placing increasing demands on databases and their management. When executing SQL (Structured Query Language) statements against a database, the contents of these statements can be distributed to different operators for execution. These operators often generate intermediate results that require temporary caching during data processing. For example, the SORT operator needs to sort all data before outputting it uniformly. Therefore, the received input data needs to be temporarily cached as an intermediate result before outputting it.

[0003] In related technologies, the temporary storage component within the database only supports storing data in a specific format. Therefore, when operators store intermediate results through the temporary storage module, they can only store them in a specific format, resulting in low access efficiency for the intermediate results of operators in non-specific formats.

[0004] Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide a data processing method and apparatus, an electronic device, and a storage medium.

[0006] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions.

[0007] According to a first aspect of one or more embodiments of this specification, a data processing method is proposed, the method comprising: in response to an operator generating an intermediate result during the execution of an SQL statement, controlling the operator to call a temporary storage module through a corresponding format interface, so as to store the intermediate result in a format corresponding to the operator through the temporary storage module; wherein the temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator that calls the temporary storage module to store data in different formats.

[0008] In a possible embodiment of the present specification, controlling the operator to store the intermediate result in a format corresponding to the operator through the temporary storage module includes: controlling the operator to store the intermediate result in a format corresponding to the operator in a hierarchical storage form through the temporary storage module; wherein the hierarchical storage form includes: storing data in a data block, constructing index information for each data block, and storing the index information of each data block in an index block.

[0009] In a possible embodiment of the present specification, the hierarchical storage format further includes: constructing index information for each index block, and storing the index information of each index block.

[0010] In a possible embodiment of the present specification, the data block includes at least one of the following information: block type, block number, data amount, original length of data, and data writing position.

[0011] In a possible embodiment of the present specification, the index information includes at least one of the following information: the indexed block number, the indexed block category, the hardware where the indexed block is located, and the compressed length of the indexed block.

[0012] In a possible embodiment of the present specification, the index block includes at least one of the following information: block category, index information amount, and index information.

[0013] In a possible embodiment of the present specification, controlling the operator to store the intermediate result in a hierarchical storage form according to the format corresponding to the operator through the temporary storage module includes: controlling the operator to write the intermediate result into the current data block in the memory according to the format corresponding to the operator, wherein the current data block includes the data block currently read and written by the operator; in response to the current data block being full, controlling the operator to allocate a new data block through the temporary storage module, and writing the intermediate result into the new data block according to the format corresponding to the operator.

[0014] In a possible embodiment of the present specification, controlling the operator to allocate a new data block through the temporary storage module includes: in response to the space occupied by the operator data block in the memory reaching the space threshold corresponding to the operator, controlling the operator to dump the operator data block to disk, build index information of the operator data block, and allocate a new data block in the memory through the temporary storage module; in response to the space occupied by the operator data block in the memory not reaching the space threshold corresponding to the operator, controlling the operator to allocate a new data block in the memory through the temporary storage module; wherein, the operator data block includes a data block in the memory that stores an intermediate result generated by the operator.

[0015] In a possible embodiment of the present specification, the method further includes: controlling the operator to read the target data in the current data block of the memory; in response to the current data block not containing the target data, controlling the operator to determine the data block where the target data is located in the index block through the temporary storage module, and controlling the operator to read the target data in the data block where the target data is located.

[0016] In a possible embodiment of the present specification, the method further includes: in response to the data block where the target data is located being in a disk, dumping the data block where the target data is located into a memory.

[0017] In a possible embodiment of the present specification, the method further includes: in response to the target data being distributed in multiple data blocks, controlling the operator to keep the multiple data blocks in the memory through the temporary storage module during the process of reading the target data.

[0018] In a possible embodiment of the present specification, controlling the operator to store the intermediate result in a format corresponding to the operator through the temporary storage module includes: controlling the operator to compress and store the intermediate result in a format corresponding to the operator through the temporary storage module.

[0019] In a possible embodiment of the present specification, the method further includes: controlling the temporary storage module to determine and update in real time the storage status of the intermediate result of the operator.

[0020] According to a second aspect of one or more embodiments of this specification, a data processing device is proposed, comprising: a storage module for generating intermediate results in response to an operator executing an SQL statement, controlling the operator to call a temporary storage module through a corresponding format interface, so as to store the intermediate results in a format corresponding to the operator through the temporary storage module; wherein the temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator that calls the temporary storage module to store data in different formats.

[0021] In a possible embodiment of the present specification, the storage module is used to: control the operator to store the intermediate result in a hierarchical storage form in a format corresponding to the operator through the temporary storage module; wherein the hierarchical storage form includes: storing data in a data block, constructing index information for each data block, and storing the index information of each data block in the index block.

[0022] In a possible embodiment of the present specification, the hierarchical storage format further includes: constructing index information for each index block, and storing the index information of each index block.

[0023] In a possible embodiment of the present specification, the data block includes at least one of the following information: block type, block number, data amount, original length of data, and data writing position.

[0024] In a possible embodiment of the present specification, the index information includes at least one of the following information: the indexed block number, the indexed block category, the hardware where the indexed block is located, and the compressed length of the indexed block.

[0025] In a possible embodiment of the present specification, the index block includes at least one of the following information: block category, index information amount, and index information.

[0026] In a possible embodiment of the present specification, the storage module is used to control the operator to store the intermediate result in a hierarchical storage form in a format corresponding to the operator through the temporary storage module, and is used to: control the operator to write the intermediate result in a current data block in the memory in a format corresponding to the operator, wherein the current data block includes the data block currently read and written by the operator; in response to the current data block being full, control the operator to allocate a new data block through the temporary storage module, and write the intermediate result in the new data block in a format corresponding to the operator.

[0027] In a possible embodiment of the present specification, the storage module is used to control the operator to allocate a new data block through the temporary storage module, and is used to: in response to the space occupied by the operator data block in the memory reaching the space threshold corresponding to the operator, control the operator to dump the operator data block to the disk, build index information of the operator data block, and allocate a new data block in the memory through the temporary storage module; in response to the space occupied by the operator data block in the memory not reaching the space threshold corresponding to the operator, control the operator to allocate a new data block in the memory through the temporary storage module; wherein the operator data block includes the data block in the memory that stores the intermediate result generated by the operator.

[0028] In a possible embodiment of the present specification, the device further includes a reading module, configured to: control the operator to read the target data within the current data block of the memory; in response to the current data block not containing the target data, control the operator to determine the data block where the target data is located in the index block through the temporary storage module, and control the operator to read the target data within the data block where the target data is located.

[0029] In a possible embodiment of the present specification, the apparatus further includes a loading module configured to: in response to the data block where the target data is located being in the disk, dump the data block where the target data is located into a memory.

[0030] In a possible embodiment of the present specification, the device further includes a cross-block reading module for controlling the operator to keep the multiple data blocks in the memory through the temporary storage module during the process of reading the target data in response to the target data being distributed in multiple data blocks.

[0031] In a possible embodiment of the present specification, the storage module is used to control the operator to compress and store the intermediate result in a format corresponding to the operator through the temporary storage module.

[0032] In a possible embodiment of the present specification, the apparatus further includes a recording module configured to control the temporary storage module to determine and update in real time the storage status of the intermediate result of the operator.

[0033] According to the third aspect of one or more embodiments of this specification, a database management system is proposed, which includes a temporary storage module, which is configured with multiple format interfaces. Different format interfaces are used to enable operators that call the temporary storage module to store data in different formats.

[0034] In a possible embodiment of the present specification, the temporary storage module is used to store data in a hierarchical storage format; wherein the hierarchical storage format includes: storing data in data blocks, constructing index information for each data block, and storing the index information of each data block in the index block.

[0035] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including: a processor;

[0036] A memory for storing processor-executable instructions; wherein the processor implements the method according to the first aspect by executing the executable instructions.

[0037] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0038] The technical solutions provided by the embodiments of this specification may include the following beneficial effects: the data processing method provided by the embodiments of this specification can generate intermediate results in response to an operator executing an SQL statement, control the operator to call a temporary storage module through a corresponding format interface, and store the intermediate results in a format corresponding to the operator through the temporary storage module; wherein the temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator calling the temporary storage module to store data in different formats. In other words, in this method, the operator can call the temporary storage module through different format interfaces to customize different data formats, so that the intermediate results can be stored in the format corresponding to the operator, meeting the format requirements of different operators and improving the storage efficiency of the intermediate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a flowchart of a data processing method provided by an exemplary embodiment.

[0040] FIG2 is a schematic diagram of a temporary storage module provided by an exemplary embodiment.

[0041] FIG3 is a schematic diagram of a hierarchical structure formed by a hierarchical storage format provided by an exemplary embodiment.

[0042] FIG. 4 is a schematic diagram of a data writing process provided by an exemplary embodiment.

[0043] FIG5 is a schematic diagram of a data reading process provided by an exemplary embodiment.

[0044] FIG6 is a schematic structural diagram of a device provided by an exemplary embodiment.

[0045] FIG7 is a block diagram of a data processing device provided by yet another exemplary embodiment. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0047] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0048] With the rapid development of the internet and information technology, data generation is exploding, placing increasing demands on databases and their management. When executing SQL statements against a database, the contents of these statements can be distributed to different operators for execution. These operators often generate intermediate results that require temporary caching during data processing. For example, the SORT operator needs to sort all data before outputting it uniformly. Therefore, the received input data needs to be temporarily cached as an intermediate result before output.

[0049] In related technologies, the temporary storage component within the database only supports storing data in a specific format. Therefore, when operators store intermediate results through the temporary storage module, they can only store them in a specific format, resulting in low access efficiency for the intermediate results of operators in non-specific formats.

[0050] Based on this, on the first aspect, at least one embodiment of this specification provides a data processing method that can support the format requirements of different operators. When an operator executes an SQL statement and generates intermediate results that need to be temporarily cached, the intermediate results are stored in the format corresponding to the operator, thereby improving the efficiency of the operator in storing and reading the intermediate results. At the same time, it can avoid the intermediate results from being damaged due to format conversion, that is, the stability and security of the intermediate results are improved.

[0051] Exemplarily, the method may be applied to a database management system, that is, the method may be executed by the database management system.

[0052] Please refer to FIG1 , which exemplarily shows the flow of the data processing method, including step S101 .

[0053] In step S101, in response to the intermediate results generated by the operator in the process of executing the SQL statement, the operator is controlled to call the temporary storage module (TempStore) through the corresponding format interface to (control the operator) store the intermediate results in the format corresponding to the operator through the temporary storage module.

[0054] When executing a SQL statement, the database management system can split it and distribute the split results to different operators for execution. This step occurs after the operator is assigned the entire SQL statement or the split results of the SQL statement. Intermediate results can be input data or preliminary processing results of the input data. The purpose of storing intermediate results is to enable the operator to process the intermediate results and generate output after receiving all the input data.

[0055] The database management system is configured with a temporary storage module, which is used to interact with storage media such as memory and disk. Operators can perform operations such as storing and reading intermediate results in the storage medium by calling the temporary storage module. The temporary storage module is not bound to a data format and is used to write data written by the operator that calls it to the storage medium, or to feed back the required read data to the operator that calls it. The temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable operators that call the temporary storage module to store data in different formats; that is, operators can customize different temporary data formats to the temporary storage module through different format interfaces. When an operator calls the temporary storage module through a certain format interface, the format customization is completed, and the operator can then write data to the temporary storage module in that format. For example, the temporary storage module can be configured with a row storage format interface, a column storage format interface, etc. After an operator calls the temporary storage module through the row storage format interface, the temporary storage module can write the intermediate result in the row storage format to the storage medium; after an operator calls the temporary storage module through the column storage format interface, the temporary storage module can write the intermediate result in the column storage format to the storage medium.

[0056] Please refer to Figure 2, which exemplarily shows a schematic diagram of a temporary storage module (TempStore). The temporary storage module can be customized in any format, such as Row Store, Column Store, Random Access Row Store, Encoding Row Store, etc.

[0057] Preferably, referring to FIG2 , the temporary storage module can be configured with a memory control function. The storage medium includes memory and disk; when the operator stores intermediate results through the temporary storage module, the intermediate results can be written to the memory first, and when the space occupied by the intermediate results generated by the operator in the memory reaches the space threshold corresponding to the operator, the intermediate results generated by the operator are dumped to the disk.

[0058] Preferably, please refer to Figure 2, the temporary storage module can be configured with a data compression function, which can meet the requirements of space utilization in scenarios such as indexing scenarios. That is, when (the management system of the database) controls the operator to store the intermediate results in the format corresponding to the operator through the temporary storage module, it can control the operator to compress and store the intermediate results in the format corresponding to the operator through the temporary storage module. It should be understood that the operator can compress the intermediate results through the temporary storage module when writing the intermediate results to the memory, or compress the intermediate results through the temporary storage module when transferring the intermediate results to the storage disk. This preferred example enables the data management system configured with the temporary storage module to reduce the space occupancy by compression when a certain operator generates a large number of intermediate results, thereby improving the utilization of the space in the storage medium and ensuring that the intermediate results of the operator can be stored smoothly.

[0059] Preferably, referring to FIG2 , the temporary storage module can be configured with a monitoring and diagnostic function. That is, (the database management system) can control the temporary storage module to determine and update the storage status of the intermediate results of the operator in real time, that is, determine the storage status of the intermediate results of each operator. The storage status may include the space threshold corresponding to the operator, the actual space occupied by the intermediate results generated by the operator in the memory, the actual space occupied by the intermediate results generated by the operator on the disk, etc. (The database management system) can also determine the execution status of SQL statements, memory usage, etc. based on the storage status of the intermediate results of each operator, and make corrections when abnormalities occur in the above situations.

[0060] Preferably, referring to FIG. 2 , the temporary storage module may be configured with a data IO function, through which the operator can write and read data.

[0061] The data processing method provided in the embodiments of this specification can generate an intermediate result in response to an operator executing an SQL statement, and control the operator to call a temporary storage module through a corresponding format interface, so that the temporary storage module can store the intermediate result in the format corresponding to the operator; wherein the temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator calling the temporary storage module to store data in different formats. In other words, in this method, the operator can call the temporary storage module through different format interfaces to customize different data formats, so that the intermediate result can be stored in the format corresponding to the operator, meeting the format requirements of different operators and the requirements of different scenarios, and improving the storage efficiency of the intermediate results.

[0062] Continuing with reference to FIG. 2 , the temporary storage module in some embodiments of this specification further includes a data block management function and an inter-block random access function, both of which can be implemented via hierarchical storage. Specifically, controlling the operator to store the intermediate result in a format corresponding to the operator via the temporary storage module may include controlling the operator to store the intermediate result in a format corresponding to the operator via the temporary storage module in a hierarchical storage format.

[0063] The hierarchical storage method includes storing data in data blocks, constructing index information for each data block, and storing the index information for each data block in the index block. In other words, data is stored sequentially in multiple data blocks. When a data block is full, index information is constructed for that data block and then stored in the next database. If the number of data blocks storing data exceeds one, the index information for these data blocks can be stored in a single index block.

[0064] Furthermore, the hierarchical storage format may further include: constructing index information for each index block and storing the index information of each index block. It should be understood that the index information of each index block may also be stored in another index block.

[0065] In fact, data blocks and index blocks are both macroblocks in memory or disk, etc., and they are named differently due to the different objects they store.

[0066] Please refer to Figure 3, which exemplifies the hierarchical structure of data blocks and index blocks formed by the hierarchical storage format. In the figure, Block represents the data block, Index Block represents the index block, and Block Index represents the index information. The hierarchical structure is a multi-layer tree structure. The root node can be an index block, which stores the index information of at least one index block, and the end node is a plurality of data blocks. There is at least one layer of index block nodes between each end node and the root node. The parent node of the index block node is the index block node or the root node, and the child node of the index block node is the index block node or the end node. In Figure 3, there is a layer of index block nodes between the root node and the end node. It should be understood that the root node can be stored in the memory, while the other nodes can be stored on the disk.

[0067] Please continue to refer to Figure 3. The data block includes at least one of the following information: block category magic, block number block_id, data amount cnt, data original length raw_size, and data write location payload. Among them, the block category magic, block number block_id, data amount cnt, and data original length raw_size constitute the block header of the data block. The block category magic is used to verify the data and distinguish different types of blocks; the block numbers block_id of different data blocks increase in the order of writing; the data amount cnt can be the number of data, such as the number of rows, etc. The sum of the block number block_id and the data amount cnt is the block number block_id of the next data block; the data original length raw_size is the length of the data written in the data block before compression; the data is written in the data write location payload.

[0068] Continuing with FIG3 , the index information includes at least one of the following: the indexed block number block_id, the indexed block type is_idx_block, the hardware location on_disk where the indexed block resides, the disk offset or memory pointer offset / point, and the compressed length of the indexed block length. The indexed block type is_idx_block can be a data block or an index block; the hardware location on_disk where the indexed block resides can be a disk or memory.

[0069] Continuing with FIG3 , the index block includes at least one of the following information: block type magic, index information amount cnt, and index information. The block type magic is used for data verification and differentiation of different types of blocks; the index information amount cnt can be the number of block indexes within the index block.

[0070] Exemplarily, controlling the operator to store the intermediate result in a hierarchical storage format according to a format corresponding to the operator through the temporary storage module may include:

[0071] First, the operator is controlled to write the intermediate result into the current data block in the memory according to the format corresponding to the operator, wherein the current data block includes the data block currently read and written by the operator.

[0072] Next, in response to the current data block being full, the operator is controlled to allocate a new data block through the temporary storage module, and the intermediate result is written into the new data block in the format corresponding to the operator. For example, in response to the space occupied by the operator data block in the memory reaching the space threshold corresponding to the operator, the operator is controlled to dump the operator data block to the disk, construct the index information of the operator data block, and allocate a new data block in the memory through the temporary storage module; in response to the space occupied by the operator data block in the memory not reaching the space threshold corresponding to the operator, the operator is controlled to allocate a new data block in the memory through the temporary storage module; wherein, the operator data block includes the data block in the memory that stores the intermediate result generated by the operator.

[0073] Please refer to FIG4 , which takes the row storage format as an example to describe the process of the preferred example in detail.

[0074] First, calculate the required memory space before writing according to the row storage format, that is, the memory space required for each row of data.

[0075] Then, determine whether the current block (i.e., the current data block) can be written: if the memory space of the current block is larger than the memory space required for each row of data, the current block can be written; if the memory space of the current block is not larger than the memory space required for each row of data, the current block cannot be written.

[0076] Then, if the current block can be written, write data to the current block.

[0077] Then, if the current block cannot be written, determine whether the data needs to be flushed to disk: if the space occupied by the data block storing the intermediate results generated by the operator in memory reaches the space threshold of the operator, the data needs to be flushed to disk; if the space occupied by the data block storing the intermediate results generated by the operator in memory does not reach the space threshold of the operator, the data does not need to be flushed to disk.

[0078] Then, if the data does not need to be written to disk, a new block is allocated in memory and data is written to the new block.

[0079] Finally, if disk flushing is required, the data block storing the intermediate results of the operator in memory is flushed to disk, an index is constructed, a new block is allocated in memory, and data is written to the new block.

[0080] As another example, after controlling the operator to store the intermediate result in a hierarchical storage format according to the format corresponding to the operator through the temporary storage module, the intermediate result may be randomly read in the following manner:

[0081] First, the operator is controlled to read target data in the current data block of the memory.

[0082] Next, in response to the current data block not containing the target data, the operator is controlled to determine the data block where the target data is located in the index block through the temporary storage module, and the operator is controlled to read the target data in the data block where the target data is located.

[0083] It should be understood that in response to the data block where the target data is located being in the disk, the data block where the target data is located can be dumped into the memory to facilitate data reading.

[0084] It should be understood that, in response to the target data being distributed across multiple data blocks, the operator can be controlled to retain the multiple data blocks in the memory via the temporary storage module during the process of reading the target data. For example, by controlling the lifecycle of the read data blocks via age and holder, the memory is not prematurely released for tasks that read across blocks, thereby ensuring normal execution of the task.

[0085] Please refer to FIG5 , which takes the row storage format as an example to describe the process of the preferred example in detail.

[0086] First, determine whether the row to be read exists in the current block (i.e., the current data block). If so, read the data.

[0087] Then, if the row to be read does not exist in the current block (ie, the current data block), the index is searched to determine whether the block (where the row is located) is in the memory. If so, the block (where the row is located) is searched to read the data.

[0088] Finally, if the block where the row is located is not in the memory, the disk data is loaded into the memory (that is, the block where the row is located is loaded from the disk to the memory), and then the block where the row is located is searched in the memory to read the data.

[0089] In this embodiment, the temporary storage module can support sequential access and random access through the form of hierarchical storage, thereby adapting to the random access requirements of operators such as window functions, further improving the applicability of the temporary storage module, and improving the management performance of the database management system with the temporary storage module.

[0090] FIG6 is a schematic structural diagram of a device provided by an exemplary embodiment. Referring to FIG6 , at the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0091] Referring to FIG7 , a data processing device can be applied to the device shown in FIG6 to implement the technical solution of this specification. The device includes: a storage module 701 for generating intermediate results in response to an operator executing an SQL statement, controlling the operator to call a temporary storage module through a corresponding format interface, so that the temporary storage module stores the intermediate results in a format corresponding to the operator;

[0092] The temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator that calls the temporary storage module to store data in different formats.

[0093] In a possible embodiment of the present specification, the storage module is used to: control the operator to store the intermediate result in a hierarchical storage form in a format corresponding to the operator through the temporary storage module; wherein the hierarchical storage form includes: storing data in a data block, constructing index information for each data block, and storing the index information of each data block in the index block.

[0094] In a possible embodiment of the present specification, the hierarchical storage format further includes: constructing index information for each index block, and storing the index information of each index block.

[0095] In a possible embodiment of the present specification, the data block includes at least one of the following information: block type, block number, data amount, original length of data, and data writing position.

[0096] In a possible embodiment of the present specification, the index information includes at least one of the following information: the indexed block number, the indexed block category, the hardware where the indexed block is located, and the compressed length of the indexed block.

[0097] In a possible embodiment of the present specification, the index block includes at least one of the following information: block category, index information amount, and index information.

[0098] In a possible embodiment of the present specification, the storage module is used to control the operator to store the intermediate result in a hierarchical storage form in a format corresponding to the operator through the temporary storage module, and is used to: control the operator to write the intermediate result in a current data block in the memory in a format corresponding to the operator, wherein the current data block includes the data block currently read and written by the operator; in response to the current data block being full, control the operator to allocate a new data block through the temporary storage module, and write the intermediate result in the new data block in a format corresponding to the operator.

[0099] In a possible embodiment of the present specification, the storage module is used to control the operator to allocate a new data block through the temporary storage module, and is used to: in response to the space occupied by the operator data block in the memory reaching the space threshold corresponding to the operator, control the operator to dump the operator data block to the disk, build index information of the operator data block, and allocate a new data block in the memory through the temporary storage module; in response to the space occupied by the operator data block in the memory not reaching the space threshold corresponding to the operator, control the operator to allocate a new data block in the memory through the temporary storage module; wherein the operator data block includes the data block in the memory that stores the intermediate result generated by the operator.

[0100] In a possible embodiment of the present specification, the device further includes a reading module, configured to: control the operator to read the target data within the current data block of the memory; in response to the current data block not containing the target data, control the operator to determine the data block where the target data is located in the index block through the temporary storage module, and control the operator to read the target data within the data block where the target data is located.

[0101] In a possible embodiment of the present specification, the apparatus further includes a loading module configured to: in response to the data block where the target data is located being in the disk, dump the data block where the target data is located into a memory.

[0102] In a possible embodiment of the present specification, the device further includes a cross-block reading module for controlling the operator to keep the multiple data blocks in the memory through the temporary storage module during the process of reading the target data in response to the target data being distributed in multiple data blocks.

[0103] In a possible embodiment of the present specification, the storage module is used to control the operator to compress and store the intermediate result in a format corresponding to the operator through the temporary storage module.

[0104] In a possible embodiment of the present specification, the apparatus further includes a recording module configured to control the temporary storage module to determine and update in real time the storage status of the intermediate result of the operator.

[0105] One or more embodiments of this specification also propose a database management system, which includes a temporary storage module. The temporary storage module is configured with multiple format interfaces. Different format interfaces are used to enable operators that call the temporary storage module to store data in different formats.

[0106] In a possible embodiment of the present specification, the temporary storage module is used to store data in a hierarchical storage format; wherein the hierarchical storage format includes: storing data in data blocks, constructing index information for each data block, and storing the index information of each data block in the index block.

[0107] The database management system can be applied to the electronic device shown in FIG6 , or to other devices.

[0108] More functions and details of the temporary storage module in the data management system have been introduced in detail in the data processing method of the first aspect and will not be repeated here.

[0109] One or more embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0110] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0111] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0113] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0115] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0117] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0119] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0120] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data processing method, the method comprising: In response to an intermediate result being generated during the execution of an SQL statement by an operator, controlling the operator to call a temporary storage module through a corresponding format interface, so as to store the intermediate result in a format corresponding to the operator through the temporary storage module; The temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator that calls the temporary storage module to store data in different formats.

2. The data processing method according to claim 1, wherein the step of controlling the operator to store the intermediate result in a format corresponding to the operator through the temporary storage module comprises: Controlling the operator to store the intermediate result in a hierarchical storage form according to a format corresponding to the operator through the temporary storage module; The hierarchical storage format includes: storing data in data blocks, constructing index information for each data block, and storing the index information of each data block in the index block.

3. The data processing method according to claim 2, wherein the hierarchical storage format further comprises: Index information is constructed for each index block, and the index information of each index block is stored.

4. The data processing method according to claim 2, wherein the data block includes at least one of the following information: block type, block number, data amount, data original length, and data writing position.

5. The data processing method according to claim 2, wherein the index information includes at least one of the following information: the index block number, the index block category, the hardware where the index block is located, and the compressed length of the index block.

6. The data processing method according to claim 2, wherein the index block includes at least one of the following information: block category, index information amount, and index information.

7. The data processing method according to any one of claims 2 to 6, wherein the step of controlling the operator to store the intermediate result in a hierarchical storage form according to a format corresponding to the operator through the temporary storage module comprises: Control the operator to write the intermediate result into the current data block in the memory according to the format corresponding to the operator, wherein the current data block includes the data block currently read and written by the operator; In response to the current data block being full, the operator is controlled to allocate a new data block through the temporary storage module, and the intermediate result is written into the new data block according to a format corresponding to the operator.

8. The data processing method according to claim 7, wherein the controlling the operator to allocate a new data block through the temporary storage module comprises: In response to the space occupied by the operator data block in the memory reaching the space threshold corresponding to the operator, control the The operator dumps the operator data block to a disk through the temporary storage module, constructs index information of the operator data block, and allocates a new data block in the memory; In response to the space occupied by the operator data block in the memory not reaching the space threshold corresponding to the operator, controlling the operator to allocate a new data block in the memory through the temporary storage module; The operator data block includes a data block in the memory storing an intermediate result generated by the operator.

9. The data processing method according to claim 7, further comprising: Controlling the operator to read target data in the current data block of the memory; In response to the current data block not containing the target data, the operator is controlled to determine the data block where the target data is located in the index block through the temporary storage module, and the operator is controlled to read the target data in the data block where the target data is located.

10. The data processing method according to claim 9, further comprising: In response to the data block where the target data is located being in the disk, the data block where the target data is located is dumped into the memory.

11. The data processing method according to claim 9, further comprising: In response to the target data being distributed in a plurality of data blocks, the operator is controlled to keep the plurality of data blocks in the memory through the temporary storage module during the process of reading the target data.

12. The data processing method according to claim 1, wherein the step of controlling the operator to store the intermediate result in a format corresponding to the operator through the temporary storage module comprises: The operator is controlled to compress and store the intermediate result according to the format corresponding to the operator through the temporary storage module.

13. The data processing method according to claim 1, further comprising: The temporary storage module is controlled to determine and update the storage status of the intermediate result of the operator in real time.

14. A data processing device, comprising: A storage module, configured to generate an intermediate result in response to the operator executing the SQL statement, control the operator to call a temporary storage module through a corresponding format interface, so as to store the intermediate result in a format corresponding to the operator through the temporary storage module; The temporary storage module is configured with multiple format interfaces, and different format interfaces are used to enable the operator that calls the temporary storage module to store data in different formats.

15. A database management system, the system comprising a temporary storage module, the temporary storage module being configured with a plurality of format interfaces, wherein different format interfaces are used to enable an operator that calls the temporary storage module to store data in different formats.

16. The database management system according to claim 15, wherein the temporary storage module is used to store data in a hierarchical storage form; in, The hierarchical storage form includes: storing data in data blocks, constructing index information for each data block, and storing the index information of each data block in the index block.

17. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 13 by running the executable instructions.

18. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images