Data storage method and related equipment
By using a data cache method based on scanning operator in the data query system, the intermediate data during the data query process is solved, and the problem of large data level cache space and low query level cache hit rate in the prior art is solved, thus achieving efficient data Savings of multiplexing and storage space.
Patent Information
- Application Number
- CN202310639166.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
In the existing data caching technology, data-level cache occupies a large storage space, while query-level cache hit rate is low, making it difficult to reuse cached data.
Using a data cache method based on the scan operator, the intermediate data during the data query is stored in the cache, and data intercept and storage are performed through the identification of the scan operator, thereby improving the cache multiplexing rate and hit rate.
It reduces the consumption of computing resources during data query, improves the hit rate of cache and data multiplexing efficiency, and saves storage space.
Smart Images

Figure CN119066094A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data storage, and in particular to a data storage method and related equipment. Background Art
[0002] In data query systems, data caching technology is often used to improve data processing efficiency.
[0003] There are two common data caching methods: one is data-level data caching, which refers to directly caching the entire data table or the entire file involved in the query command. This method will take up a large storage space; the other is query-level data caching, which refers to caching the final query results of the query command. Although this method occupies less storage space, the cached data will only be used when the exact same query command is received again, resulting in a low cache hit rate and difficulty in using the cached data.
[0004] It can be seen that there is an urgent need for a data caching method that occupies less storage space and allows cached data to be easily reused. Summary of the invention
[0005] The present application provides a data storage method to provide a data caching method that occupies less storage space and allows cached data to be easily reused. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.
[0006] A first aspect of the present application provides a data storage method, the method comprising: obtaining a first data query instruction, the first data query instruction being used to obtain a first query result from a data source; when it is determined that the first data corresponding to the identifier of a first scan operator is not stored in a cache, obtaining the first data from a first data object of the data source according to the first scan operator; wherein the first scan operator is generated based on the first data query instruction, and the first scan operator describes the operation required to be performed to obtain the first data from the first data object in the data source; storing the identifier of the first scan operator and the first data in the cache; obtaining a first query result based on the first data, for example, further calculating the first data (for example, counting calculation, etc.) to obtain the first query result.
[0007] In the first aspect, the scanning operator describes the operation to be performed to obtain data from a data object in a data source, that is, generally speaking, the scanning operator does not involve the merging operation from different data objects. There is a strong correspondence between the scanning operator and the data in the data source. That is to say, in the first aspect, the data obtained after the corresponding scanning operator operation (such as filtering, etc.) is performed on a data object of the data source can be stored in the cache, such as the first data corresponding to the first scanning operator. The first data is part of the data in the first data object, and the first data can be considered as intermediate data in the data query process.
[0008] Compared with data-level cache, the amount of data required to store the first data (rather than the entire data object) in the cache is smaller; compared with query-level cache, since the data in the cache is not the final and complete query result of the query command, but an intermediate data in the query process (data corresponding to a scanning operator in the query command), for example, the intermediate data usually comes from a data object and is not merged with other data objects. Therefore, the probability of being reused by other query commands is higher, that is, the cache hit rate is higher, which effectively reduces the computing power overhead of the scanning operator related to the intermediate data, and can better save the computing resources consumed in the data query process.
[0009] In a possible implementation manner of the first aspect, the method is applied to a computing node cluster, the data source is located in a storage node cluster, the computing node cluster communicates with the storage node cluster, and the computing node cluster and the storage node cluster are deployed separately.
[0010] In a possible implementation manner of the first aspect, the cache is located in a storage node cluster.
[0011] In this possible implementation, the storage node cluster can provide a large amount of storage resources, so the cache space is less restricted, and in many scenarios, a larger cache space can be set. In this way, the larger cache space can provide sufficient storage space for accessing cache data, so that the related applications of cache data are not restricted by the size of the storage space.
[0012] In addition, when the data query system includes multiple computing node clusters, different computing node clusters can easily access the data in the cache of the storage node cluster, thereby facilitating data sharing and global data management between different devices (such as different computing node clusters), thereby improving the interaction efficiency between multi-node devices.
[0013] In a possible implementation of the first aspect, a first scanning operator indicates an identifier and a target operation of a first data object; obtaining first data from the first data object of a data source according to the first scanning operator includes: obtaining the first data object from the data source according to the identifier of the first data object indicated by the first scanning operator; and performing the target operation on the first data object to obtain the first data.
[0014] In a traditional data query process, since the first data is intermediate data, it is only generated in the data query process and deleted very quickly, and the first data will not be identified and intercepted for caching.
[0015] In this possible implementation, after identifying that the computing node cluster generates corresponding intermediate data from the first data object according to the first scanning operator, the intermediate data can be intercepted to obtain the first data.
[0016] The timing of intercepting the data in the data query process is related to the position of the target operation included in the first scanning operator in the data processing process. For example, the target operation can be located at a position closer to the data source, that is, the target operation can be a relatively early step in the processing steps of the first data object after the first data object is obtained. Alternatively, the target operation can be located at a position farther away from the data source and closer to the business side, but where the final first query result has not yet been obtained.
[0017] In a possible implementation of the first aspect, a target operation is performed on a first data object to obtain first data, including: when vectorized processing based on the first data object is detected, first processed data before the vectorized processing is obtained, the first processed data is obtained by performing the target operation on the first data object; and the first data is obtained based on the first processed data.
[0018] In this possible implementation, the target operation is located before the vectorization processing based on the first data object. Exemplarily, the target operation includes a filtering predicate.
[0019] In this possible implementation, the first data includes data for performing a target operation on the first data object, that is, the first data is part of the data in the first data object. Compared with the first data object stored in the storage node cluster, the amount of the first data is significantly reduced. Moreover, the metadata of the first data only includes the attribute information of the first data, and does not need to include the attribute information of the entire first data object. Therefore, the metadata of the first data is usually a part of the metadata information of the first data object. It can be seen that compared with the metadata of the first data object, the metadata of the first data has a smaller amount of data. When the first data is subsequently stored as cache data, the amount of query result data and metadata information that need to be stored are reduced, thereby reducing the amount of data that needs to be stored in the cache.
[0020] In a possible implementation manner of the first aspect, a format of the first data is a data storage format of a memory.
[0021] In this possible implementation, the data storage format of the memory is columnar data, which is easier to compress. In this way, when the first data is subsequently cached, the amount of data required for caching is reduced, and when the first data is transmitted, the amount of data to be transmitted is reduced.
[0022] In a possible implementation of the first aspect, a target operation is performed on a first data object to obtain first data, including: when it is detected that vectorized processing based on the first data object is completed, obtaining second processed data obtained after the vectorized processing is completed, the second processed data is obtained by performing the target operation on the first data object; and converting the second processed data into a data storage format of a memory to obtain the first data.
[0023] In this possible implementation, the first data in the data storage format of the memory obtained can bring at least two beneficial effects in the subsequent cache storage and cache data query and processing process: 1) Columnar data is easier to compress, so that when the first data is subsequently cached, the amount of data required for the cache is reduced, and when the first data is transmitted, the amount of data to be transmitted is reduced. 2) Compared with the solution of intercepting data before vectorization processing to obtain the first data, the solution of this example is closer to the data form required by the business side. In the subsequent data query process, if the computing node cluster obtains the first data in the data storage format of the memory from the cache, the final query result can be obtained with fewer processing steps. It can be seen that the first data in the data storage format of the memory is more convenient for business use.
[0024] In a possible implementation of the first aspect, performing a target operation on a first data object to obtain first data includes: when a physical plan execution result obtained by performing the target operation on the first data object is detected, obtaining the first data according to the physical plan execution result.
[0025] In this possible implementation, the execution result of the physical plan is closer to the business side than the above two examples of interception opportunities, that is, it is closer to the data form required by the business side. In the subsequent data query process, if the computing node cluster obtains the first data in the data storage format of the memory from the cache, the final query result can be obtained with fewer processing steps. It can be seen that the first data in the data storage format of the memory is more convenient for business use. However, in some scenarios, the execution result of the physical plan may be obtained after decompression, filling and aggregation, and the amount of data may be larger, which is not conducive to storage and compression.
[0026] In a possible implementation of the first aspect, the method also includes: generating an identifier of a first scanning operator according to a first data query instruction; sending a first cache query instruction carrying the identifier of the first scanning operator to a storage node cluster, the first cache query instruction being used to request a query of first data from the cache; and receiving first feedback information from the storage node cluster, the first feedback information being used to indicate that the first data is not stored in the cache.
[0027] In this possible implementation, in one example, whether the first data exists in the cache can be identified by a specified field in the first feedback information. For example, if the content of the specified field is 11, it indicates that the first data exists in the cache; and if the content of the specified field is 00, it indicates that the first data does not exist in the cache. In another example, the first feedback information may include the first address in the cache, so that the computing node cluster accesses the first address to determine whether the first data exists in the cache.
[0028] In a possible implementation of the first aspect, an identifier of a first scanning operator is generated according to a first data query instruction, including: after optimizing a logical plan according to the first data query instruction, generating an identifier of the first scanning operator according to the optimized logical plan; and sending a first cache query instruction carrying the identifier of the first scanning operator to a storage node cluster, including: before executing a physical plan according to the optimized logical plan, sending the first cache query instruction to the storage node cluster.
[0029] In this possible implementation, accurate generation parameters for generating the identifier of the first scanning operator can be obtained according to the optimized logic plan, thereby avoiding obtaining erroneous or unnecessary generation parameters according to interference information or inaccurate information in the unoptimized logic plan.
[0030] In addition, the cache query operation in this possible implementation method can be considered as a related processing operation performed according to the information of the optimized query statement of the first data query instruction before the data query task is subdivided, that is, it can be considered as an overall processing process based on the first data query instruction.
[0031] However, in the physical planning stage, it is necessary to disassemble and distribute the execution mode of the data query task of the first data query instruction instead of an overall processing process based on the first data query instruction, and the data processing granularity is smaller. Therefore, before the physical planning stage, it is determined that the first data corresponding to the identifier of the first scanning operator is not stored in the cache, and the query of the cached data can be completed on an overall processing node, avoiding the execution of an overall processing operation based on the first data query instruction in the fine-grained task processing stage. Instead, after determining that the first data corresponding to the identifier of the first scanning operator is not stored in the cache, the cached data query result can be conveniently passed to the subsequent fine-grained task processing process in the physical planning stage.
[0032] In a possible implementation of the first aspect, the first feedback information carries a first address in a cache, and the first data is not stored in the first address; storing an identifier of the first scanning operator and the first data in the cache includes: sending the identifier of the first scanning operator and the first data to a storage node cluster to instruct the storage node cluster to store the identifier of the first scanning operator and the first data at the first address.
[0033] In this possible implementation, when the storage node cluster determines that the first data is not stored in the cache and that the first data needs to be stored in the cache, it can allocate a first address to the first data in the cache and feed it back to the computing node. In this way, the storage node cluster can instruct the computing node cluster to store the identifier of the first scanning operator and the first data in the cache of the storage node cluster through the first address.
[0034] In a possible implementation of the first aspect, the method also includes: generating a second scan operator based on the first data query command, the second scan operator describing the operation required to obtain the second data from the second data object in the data source; when the second data corresponding to the identifier of the second scan operator is stored in the cache, obtaining the second data from the cache according to the identifier of the second scan operator; when the second data corresponding to the identifier of the second scan operator is not stored in the cache, obtaining the second data from the second data object of the data source according to the second scan operator.
[0035] In a possible implementation manner of the first aspect, obtaining the first query result based on the first data includes: obtaining the first query result based on the first data and the second data, for example, calculating the first data and the second data to obtain the first query result.
[0036] In this possible implementation, the second data is also an intermediate data in the data query process of the first data query instruction. In this way, the computing node cluster can obtain the data of each of the multiple scanning operators corresponding to the first data query instruction, including the first data and the second data, and then process the first data and the second data according to the data merging operator in the physical plan, etc., to obtain the final first query result.
[0037] In a possible implementation of the first aspect, the method also includes: obtaining a second data query instruction, the second data query instruction is used to obtain a second query result from a data source; when third data corresponding to an identifier of a third scan operator is stored in a cache, obtaining the third data from the cache according to the identifier of the third scan operator; wherein the third scan operator is generated based on the second data query instruction, and the third scan operator describes the operation required to be performed to obtain the third data from a third data object in the data source; and obtaining the second query result based on the third data.
[0038] In this possible implementation, during the data query process, when the identifier of the scanning operator in the data query instruction obtained by the computing node cluster matches the identifier of a scanning operator in the cache, the computing node cluster can quickly and efficiently obtain the corresponding data from the cache through the identifier of the corresponding scanning operator, without repeatedly executing the query and extraction operations of the corresponding data objects and the calculation operations of the related scanning operators. Instead, the final query result of the corresponding data query instruction can be efficiently obtained based on the corresponding data obtained in the cache, thereby improving the data query efficiency.
[0039] In a possible implementation of the first aspect, when third data corresponding to an identifier of a third scan operator is stored in the cache, the third data is obtained from the cache according to the identifier of the third scan operator, including: sending a second cache query instruction to the storage node cluster, the second cache query instruction is used to request to query the third data from the cache; receiving second feedback information from the storage node cluster, the second feedback information carries the third data, or the second feedback information carries the storage address of the third data in the cache, so that the computing node cluster obtains the third data from the storage address.
[0040] A second aspect of the present application provides a data storage method, characterized in that the method is applied to a storage node cluster, and there is a cache and a data source on the storage node cluster. The method includes: receiving an identifier of a first scan operator and first data from a computing node cluster, the first scan operator is generated based on a first data query command, the first data query instruction is received by the computing node cluster and is used to obtain a first query result from the data source, the first scan operator describes the operation required to obtain the first data from a first data object in the data source, the first data is used to obtain the first query result, and the first data corresponding to the identifier of the first scan operator is not stored in the cache of the storage node cluster; and storing the identifier of the first scan operator and the first data in the cache.
[0041] In a possible implementation manner of the second aspect, the computing node cluster and the storage node cluster are deployed separately.
[0042] In a possible implementation manner of the second aspect, a format of the first data is a data storage format of a memory.
[0043] In a possible implementation of the second aspect, the method also includes: receiving a first cache query instruction from a computing node cluster, the first cache query instruction carrying an identifier of a first scanning operator, the first cache query instruction being used to request a query for first data from the cache; and when it is determined that the first data is not stored in the cache, sending first feedback information to the computing node cluster, the first feedback information being used to indicate that the first data does not exist in the cache.
[0044] In a possible implementation of the second aspect, the first feedback information carries a first address in a cache, and the first data is not stored in the first address; storing an identifier of the first scanning operator and the first data in the cache includes: storing the identifier of the first scanning operator and the first data at the first address.
[0045] In a possible implementation of the second aspect, the method also includes: receiving a second cache query instruction from a computing node cluster, the second cache query instruction is used to request to query the cache for third data corresponding to an identifier of a third scan operator, the third scan operator is generated based on the second data query instruction, the second data query instruction is used to obtain a second query result from a data source, the third scan operator describes the operations required to obtain the third data from a third data object in the data source, and the third data is used to obtain the second query result; when it is determined that the cache stores the third data according to the identifier of the third scan operator, the third data stored in the cache is sent to the storage node cluster according to the second cache query instruction.
[0046] In a possible implementation manner of the second aspect, storing the identifier of the first scanning operator and the first data in the cache includes: storing the first data in a first directory, where the directory name of the first directory includes the identifier of the first scanning operator.
[0047] In this possible implementation, the first data may be stored in a first directory whose directory name includes an identifier of the first scanning operator. Thus, in a subsequent retrieval process, the storage node cluster can efficiently query the cache for data corresponding to the required scanning operator by matching the identifier of the scanning operator to be matched with the directory name of the directory in the cache.
[0048] In a possible implementation of the second aspect, the first directory corresponds to first metadata, the first metadata includes metadata of the first directory, the metadata of the first directory includes information of the first scanning operator, the size of the content contained in the first directory and / or the number of times the first directory is accessed.
[0049] In this possible implementation, in the subsequent data management process, the data in the first directory can be easily determined based on the metadata of the first directory, thereby facilitating the management and query of the data in the first directory. For example, whether the data in the first directory is hot data can be determined based on the number of visits to the first directory; in addition, data matching can be performed based on the information of the first scanning operator in the first directory.
[0050] In a possible implementation manner of the second aspect, when the information of the first directory is changed, the identifier of the first metadata is the first identifier; after the information of the first directory is changed, the identifier of the first metadata is the second identifier, and the second identifier is different from the first identifier.
[0051] In this possible implementation, the identifier of the first metadata may be the name of a first file used to store the first metadata.
[0052] In this way, it is possible to conveniently identify whether the first directory is being changed through the state of the name of the first file, thereby achieving concurrency control and avoiding conflicts caused by multiple computing node clusters writing data to the first directory at the same time.
[0053] For example, when the name of the first file is Info (that is, when the identifier of the first metadata is the first identifier), it means that the first directory is being changed; and when the name of the first file is Info.inflight (that is, when the identifier of the first metadata is the second identifier), it means that the first directory has been changed.
[0054] In a possible implementation of the second aspect, the first data includes multiple data shards; an identifier of a first scanning operator and the first data are stored in a cache, including: storing a target file in a first directory, and storing target metadata in first metadata of the first directory, the target file is obtained by aggregating multiple data shards, and the target metadata is obtained based on metadata of the multiple data shards.
[0055] In this possible implementation, multiple data shards are aggregated into a target file and then stored in the first directory in the cache. In this way, in the subsequent query process, the entire target file can be obtained by accessing the first directory once, without having to access each data shard separately multiple times. This avoids frequent input / output (I / O) interactive operations on each small file that may occur when multiple data shards are stored separately in multiple small files, thereby reducing the delay and resource consumption caused by the interactive operations.
[0056] In a possible implementation of the second aspect, the first data includes multiple data slices; the identifier of the first scanning operator and the first data are stored in the cache, including: storing different data slices in the first data in different subdirectories in the first directory, and storing the metadata of each data slice in the second metadata of the subdirectory where the corresponding data slice is located.
[0057] Compared with the solution that can only match the identifier of the scanning operator and read the cached data of the entire data corresponding to the identifier of the scanning operator, in this possible implementation method, the various data slices of the first data are stored in different sub-directories respectively, and in the subsequent cache data query process, the cached data can be matched and read at the granularity of the data slice identifier. The granularity of data matching is smaller, so it is convenient to read data from the cache in a more flexible way (for example, read partial data corresponding to the scanning operator), thereby realizing fuzzy matching of the scanning operator.
[0058] In a possible implementation of the second aspect, when the information of any subdirectory is changed, the identifier of the second metadata of the corresponding subdirectory is the third identifier; after the information of the corresponding subdirectory is changed, the identifier of the second metadata of the corresponding subdirectory is the fourth identifier, and the fourth identifier is different from the third identifier.
[0059] In this possible implementation, the identifier of the second metadata of any subdirectory may be the name of the second file used to store the corresponding second metadata.
[0060] In this way, it is possible to conveniently identify whether the corresponding subdirectory is being changed through the state of the name of the second file, thereby achieving concurrency control and avoiding conflicts caused by multiple computing node clusters writing data to the corresponding subdirectory at the same time.
[0061] In a third aspect, the present application provides a data storage device, which is applied to a computing node cluster, and the device has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as: an acquisition module, a processing module, and a storage module.
[0062] In a fourth aspect, the present application provides a computing node cluster, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.
[0063] The fifth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.
[0064] The sixth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.
[0065] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting a computing node cluster to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the computing node cluster. The chip system may be composed of a chip, or may include a chip and other discrete devices.
[0066] In an eighth aspect, the present application provides a data storage device, which can be applied to a storage node cluster, and has the function of implementing the method of the second aspect or any possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as: a processing module and a storage module.
[0067] In a ninth aspect of the present application, a storage node cluster is provided, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.
[0068] The tenth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.
[0069] In an eleventh aspect of the present application, there is provided a computer program product storing one or more computer executable instructions. When the computer executable instructions are executed by a processor, the processor executes a method as in the second aspect or any possible implementation of the second aspect.
[0070] In a twelfth aspect of the present application, a chip system is provided, which includes a processor for supporting a storage node cluster to implement the functions involved in the second aspect or any possible implementation of the second aspect. In a possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the storage node cluster. The chip system may be composed of a chip, or may include a chip and other discrete devices.
[0071] Among them, the technical effects brought about by the third to seventh aspects or any possible implementation methods thereof may refer to the technical effects brought about by the first aspect or the related possible implementation methods of the first aspect, and the technical effects brought about by the eighth to twelfth aspects or any possible implementation methods thereof may refer to the technical effects brought about by the second aspect or the related possible implementation methods of the second aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Figure 1a is an exemplary schematic diagram of a directed acyclic graph provided in an embodiment of the present application;
[0073] Figure 1b is an exemplary schematic diagram of a directed acyclic graph provided in an embodiment of the present application;
[0074] Figure 1c is an exemplary schematic diagram of a directed acyclic graph provided in an embodiment of the present application;
[0075] Figure 2 This is an exemplary flowchart of the relevant processing process of the data query instruction provided in the embodiment of the present application;
[0076] Figure 3a is an exemplary structural diagram of a data query system provided in an embodiment of the present application;
[0077] Figure 3b is an exemplary structural diagram of a data query system provided in an embodiment of the present application;
[0078] Figure 4It is a schematic diagram of an embodiment of a data storage method provided in an embodiment of the present application;
[0079] Figure 5 It is a schematic diagram of an embodiment of a data storage method provided in an embodiment of the present application;
[0080] Figure 6 This is an exemplary flowchart of the relevant processing process of the data query instruction provided in the embodiment of the present application;
[0081] Figure 7 It is an exemplary schematic diagram of the timing of intercepting data in a data query process and obtaining first data provided by an embodiment of the present application;
[0082] Figure 8 is an exemplary schematic diagram of a processing flow of obtaining first data according to a first scanning operator provided in an embodiment of the present application;
[0083] Fig. 9 is an exemplary schematic diagram of storage content of first data in a cache provided by an embodiment of the present application;
[0084] Fig.10a This is an exemplary schematic diagram of a cache data storage method provided in an embodiment of the present application;
[0085] Fig.10b This is an exemplary schematic diagram of a cache data storage method provided in an embodiment of the present application;
[0086] Fig.11 is an exemplary schematic diagram of the functional modules of the data query system provided in an embodiment of the present application;
[0087] Fig.12 is a schematic diagram of an embodiment of a data storage device provided in an embodiment of the present application;
[0088] Fig.13 is a schematic diagram of another embodiment of the data storage device provided in an embodiment of the present application;
[0089] Fig.14 is a schematic diagram of a structure of a computing node cluster provided in an embodiment of the present application;
[0090] Fig.15 It is a structural diagram of a storage node cluster provided in an embodiment of the present application. DETAILED DESCRIPTION
[0091] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0092] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.
[0093] Before describing a data storage method, system, and device provided in the embodiments of the present application, some concepts involved in the embodiments of the present application are first described.
[0094] (1) Data query instructions, physical plan, logical plan, scan operator, and data merge operator.
[0095] The data query instruction instructs to execute the data query task. The data query instruction includes a data query statement. The data query statement may adopt a structured query language (SQL) or other existing or subsequently developed query languages.
[0096] In a storage-computing integrated architecture or a storage-computing separated architecture, after receiving a data query instruction, the query engine can parse, optimize, and otherwise process the data query statements in the data query instruction, including a logical plan and a physical plan based on the data query instruction.
[0097] The logical plan refers to parsing the data query statement and defining the calculation required for the data query statement, but has not yet defined how to perform the calculation.
[0098] The physical plan refers to a series of operations that need to be performed in order to query data. This series of operations can form a tree-like "operation" graph. In the "operation" graph, each node represents an operation. The "operation" graph starts from the bottom node and moves upwards until it reaches the top node. The data obtained after executing the top node is the query result.
[0099] The "operation" graph can be converted into a directed acyclic graph (DAG), such as Figure 1a to Figure 1c As shown, a directed acyclic graph representing a physical plan is exemplarily shown. Figure 1a to Figure 1c The circle in the figure indicates the operation to be performed, and the line with an arrow indicates the direction of execution. Figure 1a to Figure 1c They are all executed from the bottom node in sequence until they reach the top node.
[0100] Figure 1a The physical plan represented by the directed acyclic graph shown is a relatively simple physical plan, which contains only one "operation" chain. The execution process of the physical plan is roughly as follows: obtaining data objects from the data source, performing a series of operations on the obtained data objects, and obtaining the final query results. For the concepts of data sources and data objects, please refer to the description below.
[0101] Compared to Figure 1a The directed acyclic graph shown, Figure 1b The structure of the directed acyclic graph shown is more complex, Figure 1b The physical plan represented by the directed acyclic graph shown contains two "operation" chains. The two "operation" chains will converge at a certain node. The execution process of the physical plan is roughly as follows: determine the data objects corresponding to the two operation chains from the data source, perform a series of operations on the corresponding data objects in each operation chain, and then merge the different data obtained after the two operation chains complete the processing, and finally obtain the final query result.
[0102] Compared to Figure 1a as well as Figure 1b The directed acyclic graph shown, Figure 1c The structure of the directed acyclic graph shown is more complex, Figure 1c The physical plan represented by the directed acyclic graph shown contains four "operation" chains (as marked in the figure: chain 1, chain 2, chain 3, and chain 4). Chain 1 and chain 2 will merge at a certain node; after merging, they will merge with chain 3. After merging with chain 3, they will merge with chain 4 for merging.
[0103] It can be seen that the physical plan involves two types of operators. One operator is a single operation chain. After this type of operator extracts the data object from the data source, it can perform a series of operations on the data object, such as filtering, column pruning, row pruning, mapping, aggregation, counting, etc. In the embodiment of the present application, this type of operator is called a scanning operator. The other operator is a single operation chain that needs to perform aggregation and merging operations. This type of operator can merge multiple operation chains, or aggregate one or more other operation chains after merging multiple operation chains. In the embodiment of the present application, this type of operator is called a data merging operator. Figure 1b In , the operation of merging two operation chains is called data merging operator. Figure 1c In the example, the operation of merging chain 1 and chain 2 is called a data merging operator. The operation of merging the data after merging chain 1 and chain 2 with chain 3 is also called a data merging operator.
[0104] The physical plan includes at least one scan operator. When the physical plan includes multiple scan operators, the physical plan also includes at least one data merge operator.
[0105] After the directed acyclic graph is generated, a task set of one or more tasks of one or more scanning operators may be generated according to the directed acyclic graph, and then the tasks may be assigned to executors for execution through a task scheduler.
[0106] like Figure 2 , which is an exemplary flowchart of the processing process of a data query instruction.
[0107] like Figure 2 In the example shown, in a storage-computing integrated architecture or a storage-computing separated architecture, after obtaining the data query instruction, the logical plan can be executed on the SQL in the data query instruction, and the logical plan can be optimized. For example, based on preset rules such as constant folding, predicate pushdown, pruning, Boolean expression simplification, etc., the logical plan is optimized, and the optimal logical plan is determined as the optimized logical plan. Then, the physical plan can be executed on the optimized logical plan to obtain a directed acyclic graph representing the physical plan, which includes a scanning operator and a data merging operator. Then, through the task scheduler, the task corresponding to each scanning operator is generated, and the task is assigned to the executor in the process for execution. Figure 2 In the example, tasks A, B and C corresponding to the scan operator in the directed acyclic graph are shown by way of example. The three tasks can be executed in parallel or serially by executors in different threads (such as executor A, executor B and executor C in the figure).
[0108] It can be seen that the scan operator describes the operation to be performed to obtain data from a data object in the data source, that is, generally speaking, the scan operator does not involve the merging operation from different data objects. There is a strong correspondence between the scan operator and the data in the data source. That is to say, in an embodiment of the present application, the data obtained after performing the corresponding scan operator operation (such as filtering, etc.) on a data object in the data source can be stored in the cache. Compared with the data-level cache, the amount of data in the cache is smaller; compared with the query-level cache, since the data in the cache is not the final query result of the query command, but an intermediate data in the query process (data corresponding to a scan operator in the query command), the intermediate data can be hot data in the data object in the data source, and thus the probability of being reused by other query commands is higher, that is, the cache hit rate is higher.
[0109] (2) Data source and cache store.
[0110] A data source refers to a collection of data. In a storage-computing integrated architecture, the data source can be stored in a storage device in the architecture. In a storage-computing separation architecture, the data source can be deployed in a storage node cluster. The organizational granularity of data in a data source is called a data object, and the data object can be a data table, that is, the data in the data source is organized in the form of a dimension table, and the data source includes multiple data tables. The data object can also be a file, that is, the data in the data source is organized at the granularity of a file, and the data source includes multiple files. The data object can also be unstructured data, such as jason, documents, etc. The embodiments of the present application do not limit the specific type of the data object.
[0111] In an embodiment of the present application, a cache refers to a data set that stores part of the data in a data source, and the part of the data may be data that may need to be queried later or data with a high query frequency. Specifically, the cache may include multiple entries, each of which includes an identifier of a scan operator and data corresponding to the scan operator; for example, the multiple entries include a first entry, and the first entry includes an identifier of a first scan operator and first data corresponding to the first scan operator, wherein the first scan operator describes the operation to be performed to obtain the first data from a data object (e.g., a first data object) in the data source. Therefore, it can be understood that the data corresponding to each scan operator in the cache is usually partial data of a data object in the data source; for example, in the above example, the first data is partial data of the first data object, and specifically, the first data is the data obtained after the operation of the first scan operator (filtering, clipping, etc.) is performed on the first data object. The identifier of the scan operator is used to identify the scan operator, and the identifier of the scan operator is unique, and different scan operators have different identifiers. The above entries can be expressed as a key-value pair, wherein the identifier of the scan operator is the key in the key-value pair, and the data corresponding to the scan operator is the value in the key-value pair.
[0112] The space for storing the above cache and data source can be understood as storage space.
[0113] In the storage-computing integrated architecture, the storage space can be located entirely on the node's hard disk, entirely on the node's memory, or partially on the node's memory and partially on the node's hard disk. For example, the cache is located in the memory and the data source is located on the hard disk.
[0114] In the storage-computing separation architecture, the cache and data source are usually located on the storage node cluster. Of course, the cache can also be located on the computing node cluster. In the case where both the cache and the data source are located in the storage node cluster, the embodiments of the present application do not limit the deployment location of the cache and the data source on the storage node cluster. For example, the cache and the data source can be located on different storage nodes in the storage node cluster, or part of the cache and part of the data source can be located on the same storage node, and there can also be other deployment methods.
[0115] For example, in some examples, the data source and the cache may be located on different storage nodes of the storage node cluster. For example, the storage node cluster includes storage node A, storage node B, storage node C, and storage node D, the data source is stored in storage node A, storage node B, and storage node C, and different data objects may be stored on different storage nodes, such as data object A in the data source is stored in storage node A, data object B in the data source is stored in storage node B, and data object C in the data source is stored in storage node C. The cache is stored in storage node D, for example, specifically, it can be stored in a solid state disk (SSD) in storage node D to achieve high-speed data reading. In other examples, the data source and the cache may be stored in the same storage node, but the data source is stored in the mechanical hard disk of the storage node, and the cache is stored in the SSD of the storage node.
[0116] For the convenience of explanation, in the embodiments of the present application, the process of reading data from or writing data to a storage node cluster is directly represented by reading data from or writing data to a cache or data source. It should be understood that reading data from or writing data to a cache or data source is essentially reading data from or writing data to a storage device such as a hard disk or memory.
[0117] (3) Data sharding and data shard identification.
[0118] The data object in the data source may include multiple data shards. The data shards can be understood as further divisions of the data object. The data shards are data with a smaller granularity than the data object. The division method of the data shards in the data object and the size of the data shards are related to a series of operations on the data object in the query plan. The embodiments of the present application do not limit the number of data shards included in the data object and the size of each data shard.
[0119] Therefore, for a data object, the data in the data object (such as the first data mentioned in the embodiment of the present application) may also include one or more data fragments.
[0120] The data in the data source can be updated. From the perspective of data sharding, the data in the data source is updated when some or all of the data shards included in the data in the data source are updated. The ways of updating data include but are not limited to: adding data shards, reducing data shards, and modifying data shards.
[0121] Since the data corresponding to a scan operator in the cache is part of the data of a data object in the data source, the data corresponding to a scan operator stored in the cache also includes one or more data shards. When the data in the data source does not change, the data shards included in the data in the data source and the data shards included in the data stored in the cache are consistent. If the data in the data source changes, if the data stored in the cache cannot be synchronized and updated in a timely manner, the data shards included in the data in the data source and the data shards included in the data stored in the cache may be completely inconsistent or partially inconsistent.
[0122] In addition to each data shard in the stored data, the cache also stores the sub-identifier of each data shard. The sub-identifier of each data shard in the cache is generated based on the metadata of each data shard (may be all or part of the data in the metadata). For the convenience of description, in the subsequent examples, the sub-identifier of the data shard stored in the cache is called the data shard identifier.
[0123] It should be noted here that the metadata of the data shard used to generate the sub-identifier of each data shard in the cache is the metadata of the data shard when the data shard is written to the cache. At this time, the metadata of the data shard used describes the properties of the data shard located in the data source when the data shard is written to the cache (such as the length, name, modification record, etc. of the data shard).
[0124] In the cache, each data shard and its data shard identifier can be organized in the form of a key-value pair (key value, KV), where the data shard identifier is the key (key, K) in the key-value pair, and the data shard itself is the value corresponding to the key (value, V).
[0125] In addition, in an embodiment of the present application, under the storage-computing separation architecture, the storage node provides metadata services to the computing node. The storage node can provide the metadata of each data shard in the data source to the computing node. The metadata of each data shard is used to describe the attributes of the data shard in the data source, such as name, length, modification record and other information. It should be noted that the metadata of each data shard provided by the storage node describes each data shard in the data source, not each data shard in the cache. When the data in the data source is updated, the metadata of some or all of the data shards of the data will also change. At this time, the data shards described by the metadata of some or all of the data shards of the data may be different from the data shards of the data in the cache.
[0126] In the storage-computing integrated architecture, the storage function and the computing function are concentrated in one node. The cache and the data source are located on this node. The node can directly obtain the metadata of each data shard of the data and the data shard identifier of the data. Then, it determines whether the data in the cache is consistent with the data in the data source based on the data shard identifier of the data.
[0127] The embodiments of the present application can be applied to a data query system.
[0128] like Figure 3a In the example shown, the data query system can be a storage-computing separation architecture, which refers to the separation of computing functions and storage functions. A computing node cluster with computing functions and a storage node cluster with storage functions are deployed in the system respectively. The computing node cluster undertakes the main computing function, and the storage node cluster is mainly used to store data. Figure 3a Only a data query system under the storage and computing separation architecture is shown by way of example. The data storage method provided in the embodiments of the present application is also applicable to other data query systems under the storage and computing separation architecture.
[0129] like Figure 3a In the example shown, the data query system specifically includes a storage node cluster and at least one computing node cluster, that is, the value of N in the figure can be a positive integer.
[0130] Figure 3a Any computing node cluster in may include one computing node or multiple computing nodes.
[0131] The specific type of each computing node is not limited herein. For example, each computing node can be a terminal device, a single server or a server cluster, or a virtual machine (VM) or a container.
[0132] There is no restriction on the type of terminal device.
[0133] Exemplarily, the terminal device may be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a terminal in industrial control, a terminal in self driving, a terminal in remote medical, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, a terminal in the Internet of Things (IoT), a wearable device, a robot, or a combination of one or more thereof.
[0134] In some scenarios, any computing node cluster may be a client device of any user. In this scenario, different computing node clusters in the communication system may be different client devices. Multiple client devices may access the storage node cluster.
[0135] In other scenarios, any computing node cluster may also be a cloud resource provided by a cloud platform. In this scenario, the data query system may include one or more computing node clusters. When the communication system includes multiple computing node clusters, different computing node clusters may be cloud resources purchased by different users, or may be different parts of the same cloud resource.
[0136] A storage node cluster may include one storage node or multiple storage nodes.
[0137] A storage node cluster can provide data storage services, but a storage node cluster is not limited to providing data storage services. For example, in some scenarios, a storage node cluster also has certain data processing capabilities.
[0138] The specific type and structure of the storage node cluster are not limited here.
[0139] In one example, the storage node cluster is a distributed storage system. In another example, the storage node cluster is a storage array, for example, the storage node cluster may be a storage device of a type such as a network attached storage (NAS) device.
[0140] There may be multiple specific types of each storage node, which are not limited here. Each storage node includes but is not limited to storage functions. The functions of each storage node may be the same or different. Exemplarily, the multiple storage nodes may include backup nodes, thereby achieving backup redundancy of part or all of the data in the storage node cluster, and improving the ability to continuously provide data storage services in the event of a node failure.
[0141] Furthermore, multiple storage nodes in the storage node cluster may be located in the same physical location, such as the same data center; or, may be located in different physical locations, for example, two storage nodes in the storage node cluster may be located in different data centers.
[0142] In this example, the cache can be located in a cluster of storage nodes.
[0143] The storage node cluster can provide a large amount of storage resources. Therefore, the cache space is less restricted. In many scenarios, a larger cache space can be set.
[0144] In this way, the larger cache space can provide sufficient storage space for accessing the cache data, so that the related applications of the cache data are not limited by the size of the storage space.
[0145] In addition, when the data query system includes multiple computing node clusters, different computing node clusters can easily access the data in the cache of the storage node cluster, thereby facilitating data sharing and global data management between different devices (such as different computing node clusters), thereby improving the interaction efficiency between multi-node devices.
[0146] In the embodiments of the present application, there may be many specific application scenarios involving the computing node cluster and the storage node cluster, which are not limited here.
[0147] In some scenarios, big data management can be achieved through computing node clusters and storage node clusters. In addition, in big data management, storage and computing can be separated through computing node clusters and storage node clusters, where computing node clusters are mainly used to provide data processing services, while storage node clusters are mainly used to provide data storage services.
[0148] The storage node cluster may store columnar data, which may be existing columnar data types (such as data in parquet format or data in Hive optimized row columnar (ORC) format), or other columnar data types developed later.
[0149] In other scenarios, database applications can be implemented through computing node clusters and storage node clusters.
[0150] like Figure 3b In the example shown, the data query system can also be a storage-computing integrated architecture, in which the computing node cluster and the storage node cluster are no longer distinguished, and the computing function and the storage function can be concentrated in one node. From a hardware perspective, the structure of the node can be similar to that of the computing node, for example, it can include a processor, a memory, a bus, and a communication interface. The cache can be located in the memory of the node. The node also has the functions of the storage node and the computing node mentioned above. For example, the node can receive a data query command, generate a scanning operator according to the data query command, and determine whether data is stored in the cache according to the identifier of the scanning operator (the data is the data that needs to be obtained from the data source using the scanning operator). In the case of determining that data is stored in the cache, the data is obtained from the cache, and the query results are generated using the data. In the case of determining that no data is stored in the cache, the data is obtained from the data source using the scanning operator, and the query results are generated using the data. In addition, the node can also store the data and the identifier of the scanning operator in the cache. In the embodiments of the present application, only the storage system under the storage-computing separation architecture is used as an example to illustrate the data query method. The execution method of the data storage method in the storage-computing integration architecture can refer to the execution method of the data storage method under the storage-computing separation architecture. It is only necessary to regard the operations performed by the storage node cluster and the computing node cluster as the operations performed by the nodes in the storage-computing integration architecture, and this process will not be described in detail.
[0151] The following introduces a data storage method based on the above data query system.
[0152] like Figure 4 As shown, based on the above data query system, an embodiment of the present application provides a data storage method, which includes steps 401-405.
[0153] Step 401: A computing node cluster obtains a first data query instruction.
[0154] The first data query instruction is used to obtain a first query result from a data source.
[0155] Exemplarily, the first data query instruction may adopt a structured query language, or may adopt other existing or subsequently developed query languages.
[0156] The source of the first data query instruction is not limited here.
[0157] For example, in a database scenario, the data query system is deployed with a database, and the data in the data source on the storage side is the data in the database. The computing node cluster acts as the "front end" of the database and interacts directly with the user or the device deployed on the user side. When a user needs to query the data in the database, a first data query command is triggered, so that the computing node cluster can obtain the first data query command. The embodiment of the present application does not limit the type of database, and the database can be Oracle, MySQL, or SQL Server. The first data query command triggered by the user can carry SQL indicating a data query task.
[0158] In a big data scenario, a high-performance computing node cluster is used to analyze or process a large amount of data, and a storage node cluster is used to store the data to be processed or analyzed, or the processed data. In other words, the data source includes the data to be processed or analyzed, or the processed data. When the computing node cluster needs to process or analyze the data, the computing node cluster triggers a first data query command.
[0159] The query operation indicated by the first data query instruction may also have multiple situations.
[0160] Exemplarily, the first data query instruction may be used to perform a table scan query. A table scan is a data retrieval operation that requires reading all pages in a table to find content that meets the query condition.
[0161] Of course, in other examples, the first data query instruction may also be used to execute other types of retrieval operations currently or developed later, which are not limited here.
[0162] The first query result is the final query result obtained by the first data query instruction request. The first query result can be fed back by the computing node cluster to the initiator of the first data query instruction, for example, it can be displayed on the display device of the computing node cluster, or it can be sent to the client device that generates the first data query instruction.
[0163] Step 402: The computing node cluster determines whether first data corresponding to the identifier of the first scanning operator is stored in the cache.
[0164] The first scanning operator is generated based on the first data query instruction, and the first scanning operator describes the operation required to be performed to obtain the first data from the first data object in the data source.
[0165] In the embodiment of the present application, the scanning operator indicates the identifier of the data object and the operation performed on the data object, that is, the scanning operator indicates the following information: what operation is performed on which object in the data source.
[0166] Taking the first scan operator as an example, the one or more scan operators obtained by the computing node cluster based on the first data query instruction include the first scan operator. The first scan operator indicates the identifier of the first data object and the operation performed on the first data object.
[0167] There may be multiple types and quantities of specific operations performed by the first scanning operator on the first data object, three of which are exemplarily listed below:
[0168] The first operation is the extraction of data objects, which refers to extracting data objects from the database. The second operation is column projection, which can also be called column clipping in some cases, which refers to extracting one or more columns of a data object. The third operation is the filtering predicate, which can also be called row clipping. The filtering predicate refers to filtering one or more rows of a data object to filter out one or more rows that meet the filtering conditions.
[0169] For example, assume that the SQL in the first data query command received by the computing node cluster includes SELECT x, yFROM table1 WHERE x>3AND y<5, and the first scan operator of the query plan generated based on the SQL indicates the three types of operations mentioned above, where "FROM table1" represents the first type of operation, which requires extracting a data object named table1. "SELECT x, y" represents the second type of operation, and the columns to be extracted are identified as x and y. "WHERE x>3ANDy<5" represents the third type of operation, which requires filtering the two columns x and y to filter out one or more rows that satisfy "x>3 and y<5".
[0170] Assume that the SQL in the first data query command received by the computing node cluster includes SELECT m, n FROMtable2, and the first scan operator of the query plan generated based on the SQL indicates the two types of operations mentioned above, where "FROMtable2" represents the first operation, which requires extracting a data object named table21. "SELECT m, n" represents the second operation, and the column identifiers of the columns to be extracted are m and n.
[0171] In practical applications, the first scan operator may involve operations other than the three operations mentioned above, such as aggregation, sum, or count.
[0172] From the description of the scan that may be involved in the first scan operator, it can also be seen that the first scan operator does not involve a merging operation of data from different sources (such as different data objects) in a data source.
[0173] From the perspective of the operations included in the scanning operator, the three types of operations, namely, data object extraction operations, column projection, and filter predicates, are mainly used to locate part of the data in the data object, which has not changed. If the first scanning operator includes other operations in addition to these three operations, the other operations only process the part of the data, and do not need to be combined with other data in the data source during the processing. It can be seen that the data extracted by the scanning operator has a strong correspondence with the data in the data source.
[0174] The embodiment of the present application does not limit the method for generating the identifier of the first scanning operator. The following introduces a method for generating the identifier of the first scanning operator by taking the three types of operations including data object extraction operation, column projection (Projection), and filter predicate (Predicate) in the first scanning operator as an example.
[0175] The identifier ScanKey of the first scan operator can be generated by means of a hash function (Hash), and the identifier ScanKey of the first scan operator satisfies:
[0176] ScanKey=Hash(TableName+Projections+Predicates)
[0177] Among them, TableName is the name of the data object (i.e., the first data object) to be extracted by the data object extraction operation; Projections is the column identifier of the column to be extracted in the column projection. When multiple columns need to be extracted in the column projection, Projections is the column identifier of the multiple columns; Predicates is the filtering condition that the rows to be extracted must meet in the filtering predicate. This application does not limit the specific type of hash function (Hash), and the hash function can be MD5 or SHA-1, etc.
[0178] In addition, in order to ensure the consistency of the first scan operator, the order of column identifiers in Projections can be constrained, that is, when generating the identifier of the first scan operator, the order of column identifiers input into the hash function needs to meet the set requirements, such as the set requirement that the column identifiers can be sorted in ascending order. Similarly, the order of the components of the filter conditions in Predicate can also be constrained, that is, when generating the identifier of the first scan operator, the order of the components of the filter conditions input into the hash function needs to meet the set requirements, such as the set requirement that the column identifiers can be sorted in ascending order.
[0179] For example, the data query tasks represented by the two SQL statements "SELECT x,y FROM table1 WHERE x>3AND y<5" and "SELECT y,x FROMtable1 WHERE y<5AND x>3" are the same, but the order of column identifiers (such as x, y) in the Projection and the order of the components of the filter conditions in the Predicate (such as x>3, y<5) are inconsistent. If the order of column identifiers or the order of the components of the filter conditions are not unified, the identifiers of the first scan operators generated based on the two SQL statements may be different. By unifying the order of column identifiers or the order of the components of the filter conditions, such as unifying the column identifiers to an order of x first and y last, the identifiers of the first scan operators generated based on the two SQL statements will eventually be the same.
[0180] When the first scan operator also includes other operations in addition to the above three operations, the parameters required to perform other operations and TableName, Projections, and Predicates can also be used as inputs of the hash function to obtain the identifier of the first scan operator. When the first scan operator indicates some of the above three operations, the input of the hash function includes the parameters of the partial operation (that is, the part of TableName, Projections, and Predicates).
[0181] In the present application embodiment, Figure 5 As shown, after generating the identifier of the first scanning operator, the computing node cluster determines whether the first data corresponding to the identifier of the first scanning operator is stored in the cache.
[0182] In the storage-computing separation architecture, the cache is located in the storage node cluster. In the storage-computing integration architecture, the cache can be located in the storage in the centralized architecture.
[0183] Taking the storage-computing separation architecture as an example, the storage node cluster may pre-store at least one preset scanning operator identifier, and the preset data corresponding to the at least one preset scanning operator identifier is stored in the cache.
[0184] Specifically, steps 501 - 503 may be performed to determine that the first data is not stored in the cache.
[0185] Step 501: The computing node cluster sends a first cache query instruction carrying an identifier of a first scanning operator to the storage node cluster.
[0186] The first cache query instruction is used to request to query the first data from the cache.
[0187] Step 502: The storage node cluster queries whether the first data exists in the cache according to the identifier of the first scan operator carried in the first cache query instruction.
[0188] Specifically, during the cache query process, the storage node cluster matches the identifier of the first scan operator with the identifier of at least one preset scan operator in the cache. If the identifier of the first scan operator does not match the identifier of each preset scan operator in the identifier of at least one preset scan operator, the storage node cluster determines that the first data does not exist in the cache.
[0189] Step 503: The storage node cluster sends first feedback information to the computing node cluster.
[0190] The first feedback information is used to indicate that the first data does not exist in the cache.
[0191] The specific content of the first feedback information may be in various forms.
[0192] In one example, a designated field in the first feedback information may be used to identify whether the first data exists in the cache. For example, if the content of the designated field is 11, it indicates that the first data exists in the cache; and if the content of the designated field is 00, it indicates that the first data does not exist in the cache.
[0193] In another example, the first feedback information may include a first address in the cache, so that the computing node cluster accesses the first address to determine whether the first data exists in the cache.
[0194] In one scenario of this example, if the first data exists in the cache, the first address is the storage address of the first data in the cache, and the computing node cluster can read the first data from the first address. In another scenario of this example, if the first data does not exist in the cache, the first address is the address where the first data will be stored, and after receiving the first feedback information, the computing node cluster can access the first address, but cannot read the first data from the first address, and it is determined that the first data does not exist in the cache.
[0195] In some examples, after determining that the first data does not exist in the cache, the storage node cluster may also determine whether the first data needs to be stored in the cache. If it is determined that the first data needs to be stored in the cache, the storage node cluster sends first feedback information to the computing node cluster, and the first feedback information carries the first address in the cache to instruct the computing node cluster to write the first data to the first address.
[0196] There are multiple methods for the storage node cluster to determine whether the first data needs to be stored in the cache. Exemplarily, the storage node cluster can determine whether the first data is hot data, for example, whether the number of requests to obtain the first data is greater than a preset number within a specified time period. If the first data is determined to be hot data, it can be determined that the first data needs to be stored in the cache, so that the first feedback information is sent to the computing node cluster.
[0197] In some other examples, the storage node cluster determines that the first data does not exist in the cache and determines that the first data does not need to be stored in the cache, and can send feedback information that does not carry the first address to the computing node cluster; in this way, after receiving the feedback information that does not carry the first address, the computing node cluster can determine that the first data does not exist in the cache of the storage node cluster, and there is no need to write the first data to the cache during this interaction.
[0198] In some embodiments, the execution timing of the above steps 501-503 can be as follows: Figure 6 The above steps 501-503 may be performed through a plug-in, or may be implemented through functions in software such as a computing engine of a computing node cluster.
[0199] Specifically, after optimizing the logic plan according to the first data query instruction, the computing node cluster generates an identifier of the first scanning operator according to the optimized logic plan;
[0200] Before executing the physical plan according to the optimized logical plan, the computing node cluster sends a first cache query instruction to the storage node cluster and receives first feedback information from the storage node cluster.
[0201] At this time, after obtaining the optimized logical plan, the data query statement in the first data query instruction can be parsed and optimized, and the calculation required for the data query statement is determined, so that the information required to generate the identifier of the first scanning operator can be obtained.
[0202] For example, the identifier ScanKey of the first scan operator can be generated by means of a hash function (Hash), and the identifier ScanKey of the first scan operator satisfies:
[0203] ScanKey=Hash(TableName+Projections+Predicates)
[0204] Among them, TableName is the name of the data object (i.e., the first data object) to be extracted by the data object extraction operation; Projections is the column identifier of the column to be extracted in the column projection. When multiple columns need to be extracted in the column projection, Projections is the column identifier of the multiple columns; Predicates is the filtering condition that the rows to be extracted must meet in the filtering predicate. This application does not limit the specific type of hash function (Hash), and the hash function can be MD5 or SHA-1, etc.
[0205] After optimizing the logical plan, the TableName, Projections, and Predicates of the first scan operator may be obtained, thereby generating an identifier of the first scan operator.
[0206] Then, before executing the physical plan, steps 501-503 can be performed according to the identifier of the first scan operator to determine that the first data corresponding to the identifier of the first scan operator is not stored in the cache, and the first address of the first data to be stored in the cache of the storage node cluster can be obtained.
[0207] In this way, accurate generation parameters for generating the identifier of the first scanning operator can be obtained according to the optimized logic plan, thereby avoiding obtaining erroneous or unnecessary generation parameters according to interference information or inaccurate information in the unoptimized logic plan.
[0208] For example, before optimization, there may be parameters to be trimmed or merged in the logical plan. After optimization, the parameters to be trimmed can be trimmed, and the parameters to be merged can be merged, so as to obtain accurate and necessary generation parameters, and generate the identifier of the first scan operator based on the generation parameters to accurately describe the first scan operator.
[0209] In addition, steps 501-503 can be considered as related processing operations performed according to the information of the optimized query statement of the first data query instruction before the data query task is subdivided, that is, they can be considered as an overall processing process based on the first data query instruction.
[0210] However, in the physical planning stage, it is necessary to disassemble and distribute the execution mode of the data query task of the first data query instruction instead of an overall processing process based on the first data query instruction, and the data processing granularity is smaller. Therefore, before the physical planning stage, it is determined that the first data corresponding to the identifier of the first scanning operator is not stored in the cache, and the query of the cached data can be completed on an overall processing node, avoiding the execution of an overall processing operation based on the first data query instruction in the fine-grained task processing stage. Instead, after determining that the first data corresponding to the identifier of the first scanning operator is not stored in the cache, the cached data query result can be conveniently passed to the subsequent fine-grained task processing process in the physical planning stage.
[0211] Step 403 : When determining that the first data corresponding to the identifier of the first scan operator is not stored in the cache, the computing node cluster obtains the first data from the first data object of the data source according to the first scan operator.
[0212] In the embodiment of the present application, the scanning operator indicates the identifier of the data object and the operation performed on the data object, that is, the scanning operator indicates the following information: what operation is performed on which object in the data source.
[0213] Taking the first scan operator as an example, the one or more scan operators obtained by the computing node cluster based on the first data query instruction include the first scan operator. The first scan operator indicates the identifier of the first data object and the target operation performed on the first data object.
[0214] In this way, the computing node cluster can obtain the first data object from the data source based on the identifier of the first data object indicated by the first scanning operator, and perform a target operation on the first data object to obtain first data. The first data is part of the data in the first data object.
[0215] There may be various situations depending on the type and quantity of specific operations performed by the first scanning operator to obtain the first data from the first data object of the data source. For example, the target operation in the first scanning operator may include one or more operations such as extraction operations on the first data object, column projection, filter predicate, aggregation, sum, or count.
[0216] In a traditional data query process, since the first data is intermediate data, it is only generated in the data query process and deleted very quickly, and the first data will not be identified and intercepted for caching.
[0217] In the embodiment of the present application, after identifying that the computing node cluster generates corresponding intermediate data from the first data object according to the first scanning operator, the intermediate data can be intercepted to obtain the first data.
[0218] Specifically, Figure 6 As shown, during the process of executing the task by the executor, the first data can be intercepted to obtain the first data.
[0219] The timing of intercepting the data in the data query process is related to the position of the target operation included in the first scanning operator in the data processing process. For example, the target operation can be located at a position closer to the data source, that is, the target operation can be a relatively early step in the processing steps of the first data object after the first data object is obtained. Alternatively, the target operation can be located at a position farther away from the data source and closer to the business side, but where the final first query result has not yet been obtained.
[0220] The timing of intercepting the data in the data query process and the method of obtaining the first data may include but are not limited to three situations. The following is an exemplary introduction to three possible interception times and corresponding processing flows.
[0221] 1. Perform data interception before vectorization processing.
[0222] In some embodiments, performing a target operation on a first data object to obtain first data includes:
[0223] When vectorized processing based on the first data object is detected, first processed data before the vectorized processing is acquired, where the first processed data is obtained by performing a target operation on the first data object;
[0224] First data is obtained according to the first processed data.
[0225] In this embodiment of the application, vectorization processing can be used to obtain vectorized data.
[0226] The specific content of the vectorization processing may be various and is not limited here.
[0227] For example, vectorization processing may include decompression of the data to restore it to vectorized form. In addition, after decompression, in some scenarios, if there are constant columns or other data that need to be filled in the corresponding table, the restored vectorized form of the data can also be filled to obtain complete data.
[0228] The target operation is located before the vectorized processing based on the first data object. Exemplarily, the target operation includes a filtering predicate.
[0229] Therefore, in an embodiment of the present application, vectorized processing based on the first data object can be used as an interception opportunity, and when the vectorized processing is detected, first processed data before the vectorized processing is obtained, and the first processed data is data obtained after performing a target operation on the first data object.
[0230] Different computing engines may detect different ways of vectorizing data query results.
[0231] For example, Figure 7 In the interception opportunity 1 shown, in the scenario where the spark computing engine is used to query columnar data in the Parquet file format, when the first data object in the Parquet file format is received and processed by the Parquet file module, it can be that when it is detected that nextBatch() is called in the VectorizedParquetRecordReader class, it is determined that the spark computing engine executes the vectorized data reading operation in the spark library, that is, the vectorized processing based on the data query result is detected.
[0232] When the vectorization process is detected, first processing data before the vectorization process may be acquired.
[0233] For example, in a scenario where the spark computing engine is used to perform big data queries on columnar data in Parquet file format, the checkEndOfRowGroup() method can be called. In this method, ParquetFileReader reads the filtered PageReadStore data through operator pushdown, and the filtered PageReadStore data can be used as the first processed data.
[0234] Among them, operator pushdown can move the predicates in fields such as the WHERE clause in the first data query instruction to a lower layer to move the filter expression as close to the data source as possible (in this example, moved to the ParquetFileReader close to the storage node cluster), so that data filtering can be performed earlier to filter out irrelevant data earlier.
[0235] It can be seen that in this example, the PageReadStore data obtained after filtering the first data object for this purpose can be intercepted during the task execution process to serve as the first processing data.
[0236] PageReadStore data is the atomic storage unit (that is, the smallest storage unit) of columnar data in Parquet format. PageReadStore data can also be called a page.
[0237] After the PageReadStore data is intercepted, the PageReadStore data may be compressed and / or corresponding metadata may be added to obtain the first data.
[0238] In this example, the specific format of the first data is not limited. For example, the first data may be in a custom format, and the first data may be written into a file such as .cache (or other suffix format). Alternatively, the first data may also be in other existing or subsequently developed file formats.
[0239] The above examples are merely illustrative of the embodiments of the present application and are not intended to be limiting.
[0240] For example, in some other examples, in a scenario where data query is performed on columnar data in Hive ORC format, the first processed data may be in the form of an atomic storage unit stream of the columnar data in Hive ORC format.
[0241] In the embodiment of the present application, the first data includes data for performing a target operation on the first data object, that is, the first data is part of the data in the first data object. Compared with the first data object stored in the storage node cluster, the data volume of the first data is significantly reduced. Moreover, the metadata of the first data only includes the attribute information of the first data, and does not need to include the attribute information of the entire first data object. Therefore, the metadata of the first data is usually a part of the metadata information of the first data object. It can be seen that compared with the metadata of the first data object, the metadata of the first data has a smaller data volume. When the first data is subsequently stored as cache data, the data volume of the query results and metadata information that need to be stored are reduced, thereby reducing the amount of data that needs to be stored in the cache.
[0242] 2. Perform data interception after vectorization processing.
[0243] In some embodiments, performing a target operation on a first data object to obtain first data includes:
[0244] When it is detected that the vectorization processing based on the first data object is completed, second processing data obtained after the vectorization processing is completed is obtained, where the second processing data is obtained by performing a target operation on the first data object;
[0245] The second processed data is converted into a data storage format of the memory to obtain the first data.
[0246] In the embodiment of the present application, the target operation includes vectorized processing based on the first data object. For example, the target operation includes a filtering predicate and also includes vectorized processing such as data decompression and / or data filling.
[0247] For example, if Figure 7 In the interception opportunity 2 shown, in the scenario of using the spark computing engine to query the columnar data in the Parquet file format, in the vectorized data reader in the spark library of the spark computing engine, when it is detected that the corresponding data is batch processed (batch processing) or non-batch processed in the VectorizedParquetRecordReader class, it is determined that the vectorized processing based on the data query result is detected, and the vectorized data columnarBatch after the vectorized processing is intercepted as the second processing data. The second processing data is columnar vectorized data.
[0248] After obtaining the second processed data, the second processed data may be converted into a data storage format of the memory to obtain the first data.
[0249] The data storage format of the memory is columnar data, and the data storage format of the memory may be a data storage format such as the current Arrow format, or may be other data storage formats to be developed subsequently.
[0250] Since the second processed data is columnar vectorized data, the second processed data can be easily converted based on the rules of the data storage format of the memory to obtain the first data in the data storage format of the memory.
[0251] The first data in the data storage format of the memory obtained in this example can bring at least two beneficial effects in the subsequent cache storage and cache data query and processing process: 1) Columnar data is easier to compress, so that when the first data is cached later, the amount of data required for the cache is reduced, and when the first data is transmitted, the amount of data to be transmitted is reduced. 2) Compared with the solution of intercepting data before vectorization processing to obtain the first data, the solution of this example is closer to the data form required by the business side. In the subsequent data query process, if the computing node cluster obtains the first data in the data storage format of the memory from the cache, the final query result can be obtained with fewer processing steps. It can be seen that the first data in the data storage format of the memory is more convenient for business use.
[0252] 3. Perform data interception after obtaining the physical plan execution results.
[0253] In some embodiments, performing a target operation on a first data object to obtain first data includes:
[0254] When a physical plan execution result obtained by executing a target operation on the first data object is detected, the first data is obtained according to the physical plan execution result.
[0255] In an embodiment of the present application, the target operation is completed when the physical plan execution is completed. The target operation may include more operations relative to the above two interception timing examples, for example, it may include multiple operations such as filtering predicates, projections, aggregations, etc.
[0256] like Figure 7 The interception timing 3 in the example, the execution result of the physical plan is closer to the business side than the above two interception timing examples, that is, it is closer to the data form required by the business side. In the subsequent data query process, if the computing node cluster obtains the first data in the data storage format of the memory from the cache, the final query result can be obtained with fewer processing steps. It can be seen that the first data in the data storage format of the memory is more convenient for business use. However, in some scenarios, the execution result of the physical plan may be obtained after decompression, filling and aggregation, and the data volume may be larger, which is not conducive to storage and compression.
[0257] Step 404: The computing node cluster obtains a first query result based on the first data.
[0258] The computing node cluster can further process the first data to obtain a first query result. For example, the computing node cluster obtains the data corresponding to all scanning operators in the physical plan of the first data query instruction, and executes a data merging operator on the data corresponding to multiple scanning operators according to the physical plan to obtain the final first query result.
[0259] The following uses the first scan operator and the second scan operator in the physical plan of the first data query instruction as an example to exemplarily introduce the method of obtaining the first query result.
[0260] In some embodiments, the method further comprises:
[0261] A second scan operator is generated based on the first data query command, where the second scan operator describes an operation required to be performed to obtain second data from a second data object in the data source.
[0262] There are two situations according to the manner of acquiring the second data by the second scanning operator.
[0263] In one case, when the second data corresponding to the identifier of the second scanning operator is stored in the cache, step 1 is performed: the second data is obtained from the cache according to the identifier of the second scanning operator.
[0264] The storage node cluster matches the identifier of the second scan operator with the identifier of at least one preset scan operator in the cache. If the identifier of the second scan operator does not match the identifier of each preset scan operator in the identifier of at least one preset scan operator, the storage node cluster determines that the second data does not exist in the cache.
[0265] At this time, obtaining the second data from the second data object of the data source according to the second scanning operator is similar to obtaining the first data from the first data object of the data source according to the first scanning operator. For details, please refer to the relevant example of obtaining the first data from the first data object of the data source according to the first scanning operator.
[0266] In another case, when the second data corresponding to the identifier of the second scanning operator is not stored in the cache, step two is performed: obtaining the second data from the second data object of the data source according to the second scanning operator.
[0267] The storage node cluster matches the identifier of the second scan operator with the identifier of at least one preset scan operator in the cache. If the identifier of the second scan operator matches the identifier of at least one preset scan operator among the identifiers of at least one preset scan operator, the storage node cluster determines that the preset data corresponding to the matching preset scan operator in the cache is the second data.
[0268] At this time, the storage node cluster may send feedback information carrying the second data, or feedback information carrying the storage address of the second data in the cache to the computing node cluster, so that the computing node cluster can obtain the second data from the cache.
[0269] Step 404 includes: obtaining a first query result based on the first data and the second data.
[0270] The second data is also intermediate data in the data query process of the first data query instruction.
[0271] In this way, the computing node cluster can obtain the data of each of the multiple scanning operators corresponding to the first data query instruction, including the first data and the second data, and then process the first data and the second data according to the data merging operator in the physical plan to obtain the final first query result.
[0272] Step 405: Store the identifier of the first scanning operator and the first data in the cache.
[0273] After storing the identifier of the first scanning operator and the first data through the cache of the storage node cluster, in the subsequent data query process, when the computing node cluster obtains a new data query instruction, if the identifier of a scanning operator in the new data query instruction matches the identifier of the first scanning operator, the computing node cluster can quickly and efficiently obtain the first data from the cache of the storage node cluster without spending a long time and more resources to re-search the first data object from each storage node of the storage node cluster and re-obtain the first data from the first object, thereby shortening the query latency, reducing resource consumption, and improving data query efficiency.
[0274] For example, Figure 8 In the example shown, the SQL in the first data query instruction includes SELECT col2, col3 FROM table1 WHERE col2>2 AND col3<30. The first data object is table1, which specifically includes 5 pages, namely p1, p2, p3, p4 and p5. The first scan operator indicates the extraction operation and the filter predicate operation on the first data object. According to "FROM table1" and "WHERE col2>2 AND col3<30", the grayscale p3 part in table1 in the third step can be obtained as the content of the first data.
[0275] Figure 8 In a related example, the data content of the first data object is as follows: Fig. 9 As shown in a, the data stored in the cache is as follows Fig. 9 As shown in b.
[0276] Compared with p1-p5 and the complete metadata of p1-p5 in the first data object, the first data stored in the cache only includes partial data in p3, and the metadata of the first data may include location information of the first data in the first data object (such as row range, offset index, column name, etc.) and attribute information of the first data such as the length of the first data, without including attribute information such as p1, p2, p4 and p5.
[0277] It can be seen that compared with the first data object stored in the storage node cluster, the data volume of the first data is significantly reduced. In addition, the metadata of the first data only includes the attribute information of the first data, and does not need to include the attribute information of the entire first data object. Therefore, the metadata of the first data is usually a part of the metadata information of the first data object. It can be seen that compared with the metadata of the first data object, the data volume of the metadata of the first data is smaller. When the first data is stored as cache data, the data volume of the query results and metadata information that need to be stored are reduced, thereby reducing the amount of data that needs to be stored in the cache.
[0278] The execution order of step 405 and step 404 is not limited here.
[0279] The following describes the identifier of the first scanning operator and possible storage methods of the first data in the cache.
[0280] In some embodiments, storing the identifier of the first scan operator and the first data in the cache includes:
[0281] The first data is stored in a first directory, where the directory name of the first directory includes an identifier of the first scanning operator.
[0282] In an embodiment of the present application, the first data can be stored in a first directory whose directory name includes the identifier of the first scanning operator. In this way, in the subsequent retrieval process, the storage node cluster can efficiently query the data corresponding to the required scanning operator in the cache by matching the identifier of the scanning operator to be matched with the directory name of the directory in the cache.
[0283] In addition, in some embodiments, the first metadata of the first directory may also be stored in the cache to facilitate subsequent data management and query.
[0284] In some embodiments, the first directory corresponds to first metadata, the first metadata includes metadata of the first directory, the metadata of the first directory includes information of the first scanning operator, the size of the content contained in the first directory and / or the number of visits to the first directory.
[0285] The first metadata may be stored in an info file associated with the first directory.
[0286] Specifically, the first metadata may include, but is not limited to, information of the first scanning operator, the size of the content contained in the first directory, and / or the number of times the first directory is accessed.
[0287] The information of the first scan operator may include parameter information for generating an identifier of the first scan operator. For example, the identifier ScanKey of the first scan operator satisfies:
[0288] ScanKey=Hash(TableName+Projections+Predicates)
[0289] Wherein, TableName is the name of the first data object; Projections is the column identifier of the column to be extracted in the column projection. When multiple columns need to be extracted in the column projection, Projections is the column identifier of the multiple columns; Predicates is the filter condition that the rows to be extracted must meet in the filter predicate. This application does not limit the specific type of hash function (Hash), and the hash function can be MD5 or SHA-1, etc.
[0290] Then, the information of the first scanning operator may include the name of the first data object, the column identifier of the column to be extracted in the column projection, and the filtering condition that the row to be extracted must satisfy in the filtering predicate.
[0291] In this way, in the subsequent data management process, the data in the first directory can be easily determined based on the metadata of the first directory, thereby facilitating the management and query of the data in the first directory. For example, whether the data in the first directory is hot data can be determined based on the number of visits to the first directory; in addition, data matching can be performed based on the information of the first scanning operator in the first directory.
[0292] Furthermore, in some examples, the state of the first directory may be identified by changing the state of the first metadata.
[0293] Specifically, in some embodiments, when the information of the first directory is changed, the identifier of the first metadata is the first identifier;
[0294] After the information of the first directory is changed, the identifier of the first metadata is the second identifier, which is different from the first identifier.
[0295] In the embodiment of the present application, the identifier of the first metadata may be the name of a first file used to store the first metadata.
[0296] In this way, it is possible to conveniently identify whether the first directory is being changed through the state of the name of the first file, thereby achieving concurrency control and avoiding conflicts caused by multiple computing node clusters writing data to the first directory at the same time.
[0297] For example, when the name of the first file is Info (that is, when the identifier of the first metadata is the first identifier), it means that the first directory is being changed; and when the name of the first file is Info.inflight (that is, when the identifier of the first metadata is the second identifier), it means that the first directory has been changed.
[0298] In some scenarios, the first data may include multiple data shards. When the first data includes multiple data shards, the manners of storing the first data in the first directory include but are not limited to the following two:
[0299] 1. Aggregate multiple data fragments in the first data into an integral file for storage;
[0300] 2. Store multiple data fragments in the first data in different subdirectories.
[0301] The following takes the first scan operator and the second scan operator included in the physical plan corresponding to the first data query instruction as an example to exemplify the above two storage methods.
[0302] 1. Aggregate multiple data fragments in the first data into an integral file for storage.
[0303] Specifically, in some embodiments, storing the identifier of the first scan operator and the first data in the cache includes:
[0304] The target file is stored in a first directory, and the target metadata is stored in the first metadata of the first directory. The target file is obtained by aggregating multiple data slices, and the target metadata is obtained based on the metadata of the multiple data slices.
[0305] The first data may include multiple data fragments, and the division method of the first data may be determined based on the execution method of the first scanning operator.
[0306] For example, in some examples, in a computing node cluster, through a physical plan regarding a first data query instruction, a first scanning operator is divided into one or more resilient distributed datasets (RDDs), any RDD can be divided into one or more partitions, each partition can be processed by a task, and when each task is executed, a data shard in the first data can be obtained.
[0307] In an embodiment of the present application, during the caching process, multiple data shards may be aggregated to obtain a target file, and the target file may be stored in a first directory. In this way, multiple data shards may be stored in the form of an integral file.
[0308] The computing node cluster may aggregate multiple data slices using computing resources of the computing node cluster to obtain a target file, and send the target file to the storage node cluster to store it in the first directory in the cache. Alternatively, the computing node cluster may send multiple data slices to the storage node cluster to aggregate multiple data slices through the storage node cluster to obtain the target file.
[0309] For example, Fig.10aAs shown, the first data includes data shard A and data shard B. In the storage node cluster, data shard A and data shard B can be aggregated into a target file and stored in the first directory.
[0310] In addition, metadata of multiple data slices may be integrated as metadata of the first directory and stored in first metadata of the first directory.
[0311] For example, Fig.10a In the example shown, the metadata of each data shard may include one or more of the following information: the name of the data shard, the length of the data shard, the modification record of the data shard, the location of the data shard in the first data object, the storage address of the data shard in the cache, etc.
[0312] In this example, multiple data shards are aggregated into a target file and then stored in the first directory in the cache. In this way, in the subsequent query process, the entire target file can be obtained by accessing the first directory once, without having to access each data shard separately multiple times. This avoids frequent input / output (I / O) interactive operations on each small file that may occur when multiple data shards are stored separately in multiple small files, thereby reducing the delay and resource consumption caused by the interactive operations.
[0313] 2. Store multiple data fragments in the first data in different subdirectories.
[0314] In some embodiments, the first data includes a plurality of data slices;
[0315] Storing the identifier of the first scanning operator and the first data in the cache includes:
[0316] Different data slices in the first data are stored in different sub-directories of the first directory, and metadata of each data slice is stored in second metadata of the sub-directory where the corresponding data slice is located.
[0317] When the first data includes multiple data slices, multiple sub-directories can be created under the first directory, and the multiple data slices can be stored through the multiple sub-directories.
[0318] Different data slices are stored in different subdirectories under the first directory.
[0319] For example, Fig.10b As shown, the first data includes data slice A and data slice B, wherein data slice A is stored in subdirectory A under the first directory, and the directory name of subdirectory A is data slice identifier A, and data slice B is stored in subdirectory B under the first directory, and the directory name of subdirectory B is data slice identifier B.
[0320] The method of generating a data shard identifier is similar to that of generating a scan operator identifier. Here, data shard A is used as an example for description.
[0321] The data shard identifier SplitKeyA of data shard A can be generated by means of a hash function (Hash), and the data shard identifier SplitKeyA satisfies:
[0322] SplitKeyA=Hash(TableNameA+ProjectionsA+PredicatesA+SplitInfoA)
[0323] Among them, TableNameA, ProjectionsA, and PredicatesA are operations involved in data shard A in the target operation of the first scan operator. The description of TableNameA, ProjectionsA, and PredicatesA can refer to the relevant content in the generation method of the identifier of the aforementioned scan operator A, which will not be repeated here. SplitInfo indicates the metadata of data shard A or part of the metadata of data shard A.
[0324] That is, when generating the data shard identifier SplitKey of data shard A, the input of the hash function includes the parameters of the operation involved in data shard A in the target operation of the first scan operator (such as TableNameA, ProjectionsA, PredicatesA), and also includes part or all of the metadata of data shard A.
[0325] Compared to a solution that can only match the identifier of the scanning operator and read cached data for the entire data corresponding to the identifier of the scanning operator, in an embodiment of the present application, each data slice of the first data is stored in a different sub-directory, and in a subsequent cache data query process, cached data matching and reading can be performed at the granularity of the data slice identifier. The granularity of data matching is smaller, so it is convenient to read data from the cache in a more flexible manner (for example, reading part of the data slice corresponding to the scanning operator), thereby realizing fuzzy matching of the scanning operator.
[0326] In some embodiments, when the information of any sub-directory is changed, the identifier of the second metadata of the corresponding sub-directory is the third identifier;
[0327] After the information of the corresponding subdirectory is changed, the identifier of the second metadata of the corresponding subdirectory is a fourth identifier, which is different from the third identifier.
[0328] The identifier of the second metadata of any subdirectory may be the name of a second file used to store the corresponding second metadata.
[0329] In this way, it is possible to conveniently identify whether the corresponding subdirectory is being changed through the state of the name of the second file, thereby achieving concurrency control and avoiding conflicts caused by multiple computing node clusters writing data to the corresponding subdirectory at the same time.
[0330] For example, when the name of the second file is Info (that is, when the identifier of the second metadata is the third identifier), it indicates that the corresponding subdirectory is being changed; and when the name of the second file is Info.inflight (that is, when the identifier of the second metadata is the fourth identifier), it indicates that the corresponding subdirectory has been changed.
[0331] Referring to the above method, the cache may store identifiers of multiple scan operators and data corresponding to each scan operator.
[0332] In the subsequent data query process, when the identifier of the scanning operator in the data query instruction obtained by the computing node cluster matches the identifier of a scanning operator in the cache, the computing node cluster can quickly and efficiently obtain the corresponding data from the cache through the identifier of the corresponding scanning operator, without repeatedly executing the query and extraction operations of the corresponding data objects and the calculation operations of the related scanning operators. Instead, it can efficiently obtain the final query result of the corresponding data query instruction based on the corresponding data obtained in the cache, thereby improving the data query efficiency.
[0333] Taking the second data query instruction as an example, an exemplary implementation method for a computing node cluster to quickly and efficiently obtain data corresponding to a scanning operator from the cache of a storage node cluster is introduced below.
[0334] In some embodiments, the method further comprises:
[0335] The computing node cluster obtains a second data query instruction, where the second data query instruction is used to obtain a second query result from a data source;
[0336] In a case where the third data corresponding to the identifier of the third scan operator is stored in the cache, the computing node cluster obtains the third data from the cache according to the identifier of the third scan operator;
[0337] A second query result is obtained based on the third data.
[0338] In the embodiment of the present application, the second data query instruction is another data query instruction obtained by the computing node cluster relative to the first data query instruction. The content of the second data query instruction can be the same as the first data query instruction, or it can be different. The second data query instruction and the first data query instruction can come from the same source (for example, triggered and generated by the same computing node cluster), or they can come from different sources. In the embodiment of the present application, the relationship, content, and acquisition time of the first data query instruction and the second data query instruction are not limited.
[0339] The third scan operator is generated based on the second data query instruction, and the third scan operator describes the operation required to be performed to obtain the third data from the third data object in the data source.
[0340] The process of acquiring the third data from the cache according to the identifier of the third scanning operator may include the following steps:
[0341] The computing node cluster sends a second cache query instruction to the storage node cluster, where the second cache query instruction is used to request to query the third data from the cache;
[0342] When the storage node cluster determines that the cache stores the third data according to the identifier of the third scanning operator, the storage node cluster sends second feedback information to the storage node cluster according to the second cache query instruction, where the second feedback information carries the third data from the cache, or the second feedback information carries the storage address of the third data in the cache, so that the computing node cluster obtains the third data from the storage address;
[0343] The computing node cluster receives the second feedback information from the storage node cluster to obtain third data.
[0344] The third data is intermediate data in the data query process of the second data query instruction.
[0345] After obtaining the third data, the computing node cluster obtains a second query result based on the third data.
[0346] In this way, the computing node cluster can further process the third data. For example, the cluster obtains the data corresponding to all scanning operators in the physical plan of the second data query instruction, and executes the data merging operator on the data corresponding to multiple scanning operators according to the physical plan to obtain the final second query result.
[0347] It can be seen that the cache can store the identifiers of multiple scan operators and the data corresponding to each scan operator.
[0348] In this way, during the data query process, when the identifier of the scanning operator in the data query instruction obtained by the computing node cluster matches the identifier of a scanning operator in the cache, the computing node cluster can quickly and efficiently obtain the corresponding data from the cache through the identifier of the corresponding scanning operator, without repeatedly executing the query and extraction operations of the corresponding data objects and the calculation operations of the related scanning operators. Instead, it can efficiently obtain the final query result of the corresponding data query instruction based on the corresponding data obtained in the cache, thereby improving the data query efficiency.
[0349] In the embodiments of the present application, under the computing separation architecture, any of the above embodiments can be implemented through multiple functional modules in the storage node cluster and the computing node cluster.
[0350] Next, from a functional perspective, combined with the above steps, the functional modules included in the data query system are introduced.
[0351] like Fig.11 As shown, a data query system provided in an embodiment of the present application is provided.
[0352] The computing node cluster of the data query system includes a query engine and an execution engine.
[0353] 1) Query engine.
[0354] The query engine is the decision center of the computing node cluster. The query engine can parse the received data query command, generate the corresponding physical plan according to the data query command, and can also decide the data query method. For example, the query engine can determine whether the data requested by the data query command needs to be read from the cache 140 of the storage side or from the data source according to the feedback from the storage side. For another example, the query engine can also determine whether the data requested by the data query command needs to be cached in the cache according to the feedback from the storage side.
[0355] In an embodiment of the present application, after receiving a data query command (such as a first data query command and a second data query command), the query engine may send a cache query instruction (such as a first cache query instruction and a second cache query instruction) to the storage side, and the cache query instruction is used to request to obtain the data corresponding to the scan operator from the cache. The query engine determines, based on the feedback from the storage side, that the data needs to be obtained from the cache of the storage side or from the data source. For example, when the storage side feedback cache stores the data, the query engine may instruct the execution engine to obtain the data directly from the cache. For another example, when the storage side feedback cache does not store the data, the query engine may instruct the execution engine to obtain the data from the data source according to the scan operator in the query plan.
[0356] When the data is not stored in the cache, the query engine can further decide whether the data obtained by the execution engine from the data source needs to be cached in the cache. For example, the query engine can decide whether the data needs to be stored in the cache based on the historical query situation of the data. For another example, the query engine can decide whether the data needs to be stored in the cache based on the data caching strategy fed back by the storage node cluster.
[0357] In the above embodiment, the query engine may execute operations related to the above steps 401 - 402 , 404 , and 501 .
[0358] 2) Execution engine.
[0359] The execution engine is the executor of the computing node cluster. The execution engine can read data from the data source and obtain data from the cache or store data in the cache. In the embodiment of the present application, the execution engine can obtain data from the cache under the instruction of the query engine, can also obtain data from the data source under the instruction of the query engine, and can also store data in the cache under the instruction of the query engine.
[0360] Among them, the execution engine reading data from the cache means that the execution engine can directly obtain data from the cache, and does not need to execute the scan operator in the process of reading data. The execution engine queries data from the data source means that the execution engine needs to execute the scan operator on the data in the data source to obtain the data. The execution engine caches data means that the execution engine can write the identifier of the scan operator and the data corresponding to the scan operator into the cache.
[0361] The execution engine can split the data query task to be executed into multiple tasks, each task is responsible for one data slice. For example, for a data query task that obtains data from a data source, the execution engine can split the data query task into multiple sub-query tasks, each sub-query task is used to obtain a data slice of the data from a data object in the data source. For another example, for a data read task that obtains data from a cache, the execution engine can split the data read task into multiple sub-read tasks, each sub-read task is used to obtain a data slice of the data in the cache. For another example, for a data cache task that stores data from a cache, the execution engine can split the data cache task into multiple sub-cache tasks, each sub-cache task is used to store a data slice in the data. The embodiments of the present application do not limit the way in which the execution engine splits tasks.
[0362] In the above embodiment, the query engine may execute operations related to the above steps 403 and 405.
[0363] The execution engine or query engine can be implemented by software or hardware. Taking the query engine as an example, the query engine is an example of a software functional unit, and the query engine may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above computing instance may be one or more.
[0364] As an example of a hardware functional unit, the query engine may be at least one computing node, or may be a part of a computing node, such as a processor in the computing node.
[0365] This application does not limit the deployment location of the execution engine and the query engine. For example, the execution engine and the query engine can be deployed on the same computing node. The execution engine and the query engine can also be deployed on different computing nodes.
[0366] The storage node cluster is deployed with a cache server, cache, and data source. For details about cache and data source, please refer to the above description and will not be repeated here.
[0367] The following is an explanation of the functional modules in the storage node cluster.
[0368] The storage node cluster includes data source, cache and cache server.
[0369] 1. Data Source
[0370] Please refer to the above introduction about the concept and application of data source, which will not be repeated here.
[0371] 2. Cache
[0372] Please refer to the above introduction about the concept of cache and related embodiments, which will not be repeated here.
[0373] 3. Cache Server
[0374] The cache server is used to manage the cache. The cache server's management of the cache includes but is not limited to one or more of the following aspects: data hit, data pre-fetch, data elimination, and data update.
[0375] Among them, data hit means that the cache server can search the cache according to the identifier of the scan operator to determine whether the data corresponding to the scan operator is stored in the cache. For example, in an embodiment of the present application, the cache server can search the cache according to the identifier of the scan operator to determine whether the data (such as data A) is stored in the cache.
[0376] In the above embodiment, the cache server may perform operations related to the above steps 502-503.
[0377] Data prefetching means that the cache server can write the data and the identifier of the scanning operator corresponding to the data into the cache in advance. The cache server can decide the time of data prefetching and the data to be prefetched, that is, the cache server can decide at what time which data and the identifier of the scanning operator corresponding to the data are written into the cache.
[0378] Data elimination means that the cache server can delete the data in the cache and the identifier of the scanning operator corresponding to the data. The cache server can decide the time of data elimination and the data to be eliminated, that is, the cache server can decide when to eliminate which data in the cache and the identifier of the scanning operator corresponding to the data.
[0379] The embodiments of the present application do not limit the manner in which the cache server performs data pre-fetching or data elimination on the cache.
[0380] Data update means that the cache server can update the identifier of the scan operator in the cache and the data corresponding to the scan operator, and the updated data in the cache is consistent with the data in the data source. When the data in the data source is updated, for example, the execution engine accesses the data in the data source and performs operations such as adding, deleting, and modifying the data, so that the data itself is updated, the cache server can also synchronously update the data in the cache. That is, the cache caches the data before the data is updated. After the data in the data source is updated, the data stored in the cache also needs to be updated accordingly.
[0381] There are many ways for the cache server to implement data updates. For example, the cache server can monitor the execution engine's operations on the data in the data source (the updates here include but are not limited to: adding, deleting, and modifying). When it is found that the execution engine updates the data in the data source, the cache server can synchronously update the data in the cache.
[0382] The cache server can be implemented by software or hardware. As an example of a software functional unit, the cache server may include code running on a storage node. As an example of a hardware functional unit, the cache server may be at least one storage node. Alternatively, the query engine may also be a part of a storage node, such as the cache server may be a processor or a network card in the storage node.
[0383] This application does not limit the deployment location of the cache server, cache, and data source. For example, the cache server, cache, and data source can be deployed on the same or multiple storage nodes. The cache server, cache, and data source can also be deployed on different storage nodes.
[0384] The embodiments of the present application have described the data storage method from multiple aspects. The following describes the data storage device applied to a computing node cluster and the data storage device applied to a storage node cluster in the present application in conjunction with the accompanying drawings.
[0385] like Fig.12 As shown, an embodiment of the present application provides a data storage device 120.
[0386] The device 120 comprises:
[0387] The acquisition module 1201 is used to acquire a first data query instruction, where the first data query instruction is used to acquire a first query result from a data source;
[0388] The processing module 1202 is used to obtain the first data from the first data object of the data source according to the first scanning operator when it is determined that the first data corresponding to the identifier of the first scanning operator is not stored in the cache; wherein the first scanning operator is generated based on the first data query instruction, and the first scanning operator describes the operation required to be performed to obtain the first data from the first data object in the data source;
[0389] The storage module 1203 is used to store the identifier of the first scanning operator and the first data in the cache;
[0390] The processing module 1202 is further configured to obtain a first query result based on the first data.
[0391] Optionally, the device 120 is applied to a computing node cluster, the data source is located in a storage node cluster, the computing node cluster communicates with the storage node cluster, and the computing node cluster and the storage node cluster are deployed separately.
[0392] Optionally, the cache is located in a cluster of storage nodes.
[0393] Optionally, the first scan operator indicates an identification and a target operation of the first data object;
[0394] The processing module 1202 is used to:
[0395] Acquire the first data object from a data source according to an identifier of the first data object indicated by the first scanning operator;
[0396] A target operation is performed on the first data object to obtain first data.
[0397] Optionally, the processing module 1202 is used to:
[0398] When vectorized processing based on the first data object is detected, first processed data before the vectorized processing is acquired, where the first processed data is obtained by performing a target operation on the first data object;
[0399] First data is obtained according to the first processed data.
[0400] Optionally, the format of the first data is a data storage format of a memory.
[0401] Optionally, the processing module 1202 is used to:
[0402] When it is detected that the vectorization processing based on the first data object is completed, second processing data obtained after the vectorization processing is completed is obtained, where the second processing data is obtained by performing a target operation on the first data object;
[0403] The second processed data is converted into a data storage format of the memory to obtain the first data.
[0404] Optionally, the processing module 1202 is used to:
[0405] When a physical plan execution result obtained by executing a target operation on the first data object is detected, the first data is obtained according to the physical plan execution result.
[0406] Optionally, the apparatus 120 further includes an interface module 1204;
[0407] The processing module 1202 is further configured to generate an identifier of a first scanning operator according to the first data query instruction;
[0408] The interface module 1204 is used for:
[0409] Sending a first cache query instruction carrying an identifier of a first scanning operator to the storage node cluster, where the first cache query instruction is used to request to query the first data from the cache;
[0410] First feedback information is received from the storage node cluster, where the first feedback information is used to indicate that the first data is not stored in the cache.
[0411] Optionally, the processing module 1202 is used to: after optimizing the logic plan according to the first data query instruction, generate an identifier of the first scanning operator according to the optimized logic plan;
[0412] The interface module 1204 is used to send a first cache query instruction to the storage node cluster before executing the physical plan according to the optimized logical plan.
[0413] Optionally, the first feedback information carries a first address in the cache, and the first data is not stored in the first address;
[0414] The storage module 1203 is used for:
[0415] The identifier of the first scan operator and the first data are sent to the storage node cluster to instruct the storage node cluster to store the identifier of the first scan operator and the first data at the first address.
[0416] Optionally, the processing module 1202 is used to:
[0417] generating a second scan operator based on the first data query command, where the second scan operator describes an operation required to be performed to obtain second data from a second data object in the data source;
[0418] When the second data corresponding to the identifier of the second scan operator is stored in the cache, obtaining the second data from the cache according to the identifier of the second scan operator;
[0419] When the second data corresponding to the identifier of the second scan operator is not stored in the cache, the second data is acquired from the second data object of the data source according to the second scan operator.
[0420] Optionally, the processing module 1202 is used to obtain a first query result based on the first data and the second data.
[0421] Optionally, the processing module 1202 is used to:
[0422] Obtain a second data query instruction, where the second data query instruction is used to obtain a second query result from a data source;
[0423] In a case where the third data corresponding to the identifier of the third scan operator is stored in the cache, the third data is obtained from the cache according to the identifier of the third scan operator; wherein the third scan operator is generated based on the second data query instruction, and the third scan operator describes the operation required to be performed to obtain the third data from the third data object in the data source;
[0424] A second query result is obtained based on the third data.
[0425] Optionally, the interface module 1204 is used to:
[0426] Sending a second cache query instruction to the storage node cluster, where the second cache query instruction is used to request to query the third data from the cache;
[0427] Second feedback information is received from the storage node cluster, where the second feedback information carries the third data, or the second feedback information carries the storage address of the third data in the cache, so that the computing node cluster obtains the third data from the storage address.
[0428] like Fig.13 As shown, an embodiment of the present application provides a data storage device 130. The device 130 is applied to a storage node cluster, and the storage node cluster has a cache and a data source.
[0429] The device 130 comprises:
[0430] The interface module 1301 is used to receive an identifier of a first scan operator and first data from a computing node cluster, where the first scan operator is generated based on a first data query command, the first data query command is received by the computing node cluster and is used to obtain a first query result from a data source, the data source is located in a storage node cluster, the first scan operator describes an operation required to be performed to obtain the first data from a first data object in the data source, the first data is used to obtain the first query result, and the first data corresponding to the identifier of the first scan operator is not stored in a cache of the storage node cluster;
[0431] The storage module 1302 is used to store the identifier of the first scanning operator and the first data in the cache.
[0432] Optionally, the computing node cluster and the storage node cluster are deployed separately.
[0433] Optionally, the format of the first data is a data storage format of a memory.
[0434] Optionally, the interface module 1301 is used to:
[0435] Receive a first cache query instruction from a computing node cluster, where the first cache query instruction carries an identifier of a first scanning operator, and the first cache query instruction is used to request to query first data from a cache;
[0436] When it is determined that the first data is not stored in the cache, first feedback information is sent to the computing node cluster, where the first feedback information is used to indicate that the first data does not exist in the cache.
[0437] Optionally, the first feedback information carries a first address in the cache, and the first data is not stored in the first address;
[0438] The storage module 1302 is used for:
[0439] An identifier of a first scan operator and first data are stored at a first address.
[0440] Optionally, the interface module 1301 is used to:
[0441] Receive a second cache query instruction from the computing node cluster, the second cache query instruction is used to request to query the cache for third data corresponding to the identifier of the third scan operator, the third scan operator is generated based on the second data query instruction, the second data query instruction is used to obtain the second query result from the data source, the third scan operator describes the operation required to be performed to obtain the third data from the third data object in the data source, and the third data is used to obtain the second query result;
[0442] When it is determined that the cache stores the third data according to the identifier of the third scanning operator, the third data stored in the cache is sent to the storage node cluster according to the second cache query instruction.
[0443] Optionally, the storage module 1302 is used to: store the first data in a first directory, and the directory name of the first directory includes an identifier of the first scanning operator.
[0444] Optionally, the first directory corresponds to first metadata, the first metadata includes metadata of the first directory, the metadata of the first directory includes information of the first scanning operator, the size of the content contained in the first directory and / or the number of times the first directory is accessed.
[0445] Optionally, when the information of the first directory is changed, the identifier of the first metadata is the first identifier;
[0446] After the information of the first directory is changed, the identifier of the first metadata is the second identifier, which is different from the first identifier.
[0447] Optionally, the first data includes multiple data fragments;
[0448] The storage module 1302 is used to store the target file in the first directory and store the target metadata in the first metadata of the first directory. The target file is obtained by aggregating multiple data slices, and the target metadata is obtained based on the metadata of the multiple data slices.
[0449] Optionally, the first data includes multiple data fragments;
[0450] The storage module 1302 is used to store different data slices in the first data in different subdirectories in the first directory, and store metadata of each data slice in second metadata of the subdirectory where the corresponding data slice is located.
[0451] Optionally, when the information of any subdirectory is changed, the identifier of the second metadata of the corresponding subdirectory is the third identifier;
[0452] After the information of the corresponding subdirectory is changed, the identifier of the second metadata of the corresponding subdirectory is a fourth identifier, which is different from the third identifier.
[0453] Fig.14 FIG. 1 is a possible logical structure diagram of a computing node cluster 14 provided in an embodiment of the present application. The computing node cluster 14 is used to implement the functions of the computing node cluster involved in any of the above embodiments. The computing node cluster 14 includes one or more computing nodes 140 ( Fig.14Only one computing node 140 is shown as an example. Any computing node 140 includes: a memory 1401, a processor 1402, a communication interface 1403 and a bus 1404. The memory 1401, the processor 1402 and the communication interface 1403 are connected to each other through the bus 1404.
[0454] The memory 1401 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1401 may store a program. When the program stored in the memory 1401 is executed by the processor 1402, the processor 1402 and the communication interface 1403 are used to perform one or more steps of the above-mentioned data storage method embodiment.
[0455] The processor 1402 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, for executing relevant programs to implement the functions required to be executed by the acquisition module, processing module, storage module and interface module in the data storage device in the above embodiment, or to execute one or more steps of the method embodiment of the present application. The steps of the method disclosed in the embodiment of the present application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1401, and the processor 1402 reads the information in the memory 1401, and executes one or more steps of the above data storage method embodiment in combination with its hardware.
[0456] The communication interface 1403 uses a transceiver device such as but not limited to a transceiver to implement communication between the computing node cluster 14 and other devices or communication networks. For example, the computing node cluster 14 can communicate with the storage node cluster 15 through the communication interface 1403 .
[0457] The bus 1404 can realize the path for transmitting information between the various components of the computing node cluster 14 (for example, the memory 1401, the processor 1402 and the communication interface 1403). The bus 1404 can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0458] In another embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the above Fig.14 The steps performed by the processor in .
[0459] In another embodiment of the present application, a computer program product is further provided, the computer program product comprising computer execution instructions, the computer execution instructions being stored in a computer readable storage medium; when the processor of the device executes the computer execution instructions, the device executes the above Fig.14 The steps performed by the processor in .
[0460] In another embodiment of the present application, a chip system is further provided, the chip system comprising a processor, the processor being used to implement the above Fig.14 In one possible design, the chip system may also include a memory for storing necessary program instructions and data. The chip system may be composed of a chip or may include a chip and other discrete devices.
[0461] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.
[0462] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0463] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0464] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0465] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0466] Fig.15 FIG. 1 is a possible logical structure diagram of a storage node cluster 15 provided in an embodiment of the present application. The storage node cluster 15 is used to implement the functions of the storage node cluster involved in any of the above embodiments. The storage node cluster 15 includes one or more storage nodes 150 ( Fig.15 Only one storage node 150 is shown as an example. Any storage node 150 includes: a memory 1501, a communication interface 1503 and a bus 1504. In addition, in some examples, the storage node 150 may also include a processor 1502 to implement certain processing and control functions. The memory 1501, the processor 1502 and the communication interface 1503 are connected to each other through the bus 1504.
[0467] The memory 1501 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1501 may store a program. When the program stored in the memory 1501 is executed by the processor 1502, the processor 1502 and the communication interface 1503 are used to perform one or more steps of the above-mentioned data storage method embodiment.
[0468] The processor 1502 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, for executing related programs to implement the functions required to be executed by the acquisition module, processing module, storage module and interface module in the data storage device in the above embodiment, or to execute one or more steps of the method embodiment of the present application. The steps of the method disclosed in the embodiment of the present application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1501, and the processor 1502 reads the information in the memory 1501, and executes one or more steps of the above data storage method embodiment in combination with its hardware.
[0469] The communication interface 1503 uses a transceiver device such as but not limited to a transceiver to implement communication between the storage node cluster 15 and other devices or communication networks. For example, it can implement communication with the storage node cluster.
[0470] For example, the storage node cluster 15 can communicate with the computing node cluster 14 through the communication interface 1503 .
[0471] The bus 1504 can realize the path for transmitting information between the various components of the storage node cluster 15 (for example, the memory 1501, the processor 1502 and the communication interface 1503). The bus 1504 can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0472] In another embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the above Fig.15 The steps performed by the processor in .
[0473] In another embodiment of the present application, a computer program product is further provided, the computer program product comprising computer execution instructions, the computer execution instructions being stored in a computer readable storage medium; when the processor of the device executes the computer execution instructions, the device executes the above Fig.15 The steps performed by the processor in .
[0474] In another embodiment of the present application, a chip system is further provided, the chip system comprising a processor, the processor being used to implement the above Fig.15 In one possible design, the chip system may also include a memory for storing necessary program instructions and data. The chip system may be composed of a chip or may include a chip and other discrete devices.
[0475] In another embodiment of the present application, a data query system is further provided. The data query system may include the computing node cluster 14 and the storage node cluster 15 described above.
[0476] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.
[0477] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this.
Claims
1. A data storage method, characterized in that: The method comprises: Obtaining a first data query instruction, where the first data query instruction is used to obtain a first query result from a data source; When it is determined that the first data corresponding to the identifier of the first scan operator is not stored in the cache, the first data is obtained from the first data object of the data source according to the first scan operator; wherein the first scan operator is generated based on the first data query instruction, and the first scan operator describes the operation required to be performed to obtain the first data from the first data object in the data source; storing the identifier of the first scanning operator and the first data in the cache; The first query result is obtained based on the first data.
2. The method according to claim 1, characterized in that The method is applied to a computing node cluster, the data source is located in a storage node cluster, the computing node cluster communicates with the storage node cluster, and the computing node cluster and the storage node cluster are deployed separately.
3. The method according to claim 2, characterized in that The cache is located in the storage node cluster.
4. The method according to claim 3, characterized in that: The first scan operator indicates an identification and a target operation of the first data object; The acquiring the first data from the first data object of the data source according to the first scanning operator includes: acquiring the first data object from the data source according to the identifier of the first data object indicated by the first scanning operator; The target operation is performed on the first data object to obtain the first data.
5. The method according to claim 4, characterized in that The performing the target operation on the first data object to obtain the first data includes: When vectorized processing based on the first data object is detected, first processed data before the vectorized processing is acquired, where the first processed data is obtained by performing the target operation on the first data object; The first data is obtained according to the first processed data.
6. The method according to claim 4, characterized in that The format of the first data is the data storage format of the memory.
7. The method according to claim 6, characterized in that The performing the target operation on the first data object to obtain the first data includes: When it is detected that the vectorization processing based on the first data object is completed, obtaining second processing data obtained after the vectorization processing is completed, where the second processing data is obtained by performing the target operation on the first data object; The second processed data is converted into a data storage format of a memory to obtain the first data.
8. The method according to claim 4, characterized in that The performing the target operation on the first data object to obtain the first data includes: When a physical plan execution result obtained by executing the target operation on the first data object is detected, the first data is obtained according to the physical plan execution result.
9. The method according to any one of claims 3 to 8, characterized in that: The method further comprises: generating an identifier of the first scanning operator according to the first data query instruction; Sending a first cache query instruction carrying an identifier of the first scan operator to the storage node cluster, where the first cache query instruction is used to request to query the first data from the cache; First feedback information is received from the storage node cluster, where the first feedback information is used to indicate that the first data is not stored in the cache.
10. The method according to claim 9, characterized in that The step of generating the identifier of the first scanning operator according to the first data query instruction includes: After performing logic plan optimization according to the first data query instruction, generating an identifier of the first scanning operator according to the optimized logic plan; The sending the first cache query instruction carrying the identifier of the first scanning operator to the storage node cluster includes: Before executing the physical plan according to the optimized logical plan, the first cache query instruction is sent to the storage node cluster.
11. The method according to claim 9 or 10, characterized in that: The first feedback information carries a first address in the cache, and the first data is not stored in the first address; The storing the identifier of the first scanning operator and the first data in the cache includes: The identifier of the first scan operator and the first data are sent to the storage node cluster to instruct the storage node cluster to store the identifier of the first scan operator and the first data at the first address.
12. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: generating a second scan operator based on the first data query command, where the second scan operator describes an operation required to be performed to obtain the second data from a second data object in the data source; In a case where the second data corresponding to the identifier of the second scan operator is stored in the cache, acquiring the second data from the cache according to the identifier of the second scan operator; In a case where the second data corresponding to the identifier of the second scan operator is not stored in the cache, the second data is acquired from the second data object of the data source according to the second scan operator.
13. The method according to claim 12, characterized in that The obtaining the first query result based on the first data includes: The first query result is obtained based on the first data and the second data.
14. The method according to any one of claims 2 to 11, characterized in that: The method further comprises: Obtaining a second data query instruction, where the second data query instruction is used to obtain a second query result from the data source; In a case where the cache stores third data corresponding to the identifier of the third scan operator, the third data is obtained from the cache according to the identifier of the third scan operator; wherein the third scan operator is generated based on the second data query instruction, and the third scan operator describes an operation required to be performed to obtain the third data from the third data object in the data source; The second query result is obtained based on the third data.
15. The method according to claim 14, characterized in that In a case where the cache stores third data corresponding to the identifier of the third scan operator, acquiring the third data from the cache according to the identifier of the third scan operator includes: Sending a second cache query instruction to the storage node cluster, where the second cache query instruction is used to request to query the third data from the cache; Receive second feedback information from the storage node cluster, where the second feedback information carries the third data, or the second feedback information carries a storage address of the third data in the cache, so that the computing node cluster obtains the third data from the storage address.
16. A data storage method, characterized in that: The method is applied to a storage node cluster, where there is a cache and a data source, and the method includes: receiving an identifier of a first scan operator and first data from a computing node cluster, where the first scan operator is generated based on a first data query command, the first data query instruction is received by the computing node cluster and is used to obtain a first query result from the data source, the first scan operator describes an operation required to be performed to obtain the first data from a first data object in the data source, the first data is used to obtain the first query result, and the cache does not store the first data corresponding to the identifier of the first scan operator; The identifier of the first scanning operator and the first data are stored in the cache.
17. The method according to claim 16, characterized in that The computing node cluster and the storage node cluster are deployed separately.
18. The method according to claim 16 or 17, characterized in that The format of the first data is the data storage format of the memory.
19. The method according to claim 16 or 17, characterized in that The method further comprises: receiving a first cache query instruction from the computing node cluster, the first cache query instruction carrying an identifier of the first scanning operator, the first cache query instruction being used to request to query the first data from the cache; When it is determined that the first data is not stored in the cache, first feedback information is sent to the computing node cluster, where the first feedback information is used to indicate that the first data does not exist in the cache.
20. The method according to claim 19, characterized in that The first feedback information carries a first address in the cache, and the first data is not stored in the first address; The storing the identifier of the first scanning operator and the first data in the cache includes: The identifier of the first scan operator and the first data are stored at the first address.
21. The method according to any one of claims 16 to 20, characterized in that: The method further comprises: receiving a second cache query instruction from the computing node cluster, the second cache query instruction being used to request querying third data corresponding to an identifier of a third scan operator from the cache, the third scan operator being generated based on the second data query instruction, the second data query instruction being used to obtain a second query result from the data source, the third scan operator describing an operation required to be performed to obtain the third data from a third data object in the data source, and the third data being used to obtain the second query result; When it is determined that the cache stores the third data according to the identifier of the third scan operator, the third data stored in the cache is sent to the storage node cluster according to the second cache query instruction.
22. The method according to any one of claims 16 to 21, characterized in that: The storing the identifier of the first scanning operator and the first data in the cache includes: The first data is stored in a first directory, where the directory name of the first directory includes an identifier of the first scanning operator.
23. The method according to claim 22, characterized in that The first directory corresponds to first metadata, which includes metadata of the first directory, and the metadata of the first directory includes information of the first scanning operator, the size of the content contained in the first directory and / or the number of times the first directory is accessed.
24. The method according to claim 23, characterized in that When the information of the first directory is changed, the identifier of the first metadata is a first identifier; After the information of the first directory is changed, the identifier of the first metadata is a second identifier, and the second identifier is different from the first identifier.
25. The method according to claim 23 or 24, characterized in that The first data includes a plurality of data slices; The storing the identifier of the first scanning operator and the first data in the cache includes: The target file is stored in the first directory, and the target metadata is stored in the first metadata of the first directory, the target file is obtained by aggregating the multiple data slices, and the target metadata is obtained based on the metadata of the multiple data slices.
26. The method according to claim 23 or 24, characterized in that The first data includes a plurality of data slices; The storing the identifier of the first scanning operator and the first data in the cache includes: Different data slices in the first data are stored in different sub-directories of the first directory, and metadata of each data slice is stored in second metadata of the sub-directory where the corresponding data slice is located.
27. The method according to claim 26, characterized in that When the information of any of the sub-directories is changed, the identifier of the second metadata of the corresponding sub-directory is the third identifier; After the information of the corresponding subdirectory is changed, the identifier of the second metadata of the corresponding subdirectory is a fourth identifier, and the fourth identifier is different from the third identifier.
28. A data storage device, characterized in that: include: An acquisition module, used to acquire a first data query instruction, where the first data query instruction is used to acquire a first query result from a data source; a processing module, configured to obtain the first data from the first data object of the data source according to the first scanning operator when it is determined that the first data corresponding to the identifier of the first scanning operator is not stored in the cache; wherein the first scanning operator is generated based on the first data query instruction, and the first scanning operator describes an operation required to be performed to obtain the first data from the first data object in the data source; A storage module, configured to store the identifier of the first scanning operator and the first data in the cache; The processing module is further configured to obtain the first query result based on the first data.
29. The device according to claim 28, characterized in that The device is applied to a computing node cluster, the data source is located in a storage node cluster, the computing node cluster communicates with the storage node cluster, and the computing node cluster and the storage node cluster are deployed separately.
30. The device according to claim 29, characterized in that The cache is located in the storage node cluster.
31. The device according to claim 30, characterized in that The first scan operator indicates an identification and a target operation of the first data object; The processing module is used for: acquiring the first data object from the data source according to the identifier of the first data object indicated by the first scanning operator; The target operation is performed on the first data object to obtain the first data.
32. A data storage device, characterized in that: The device is applied to a storage node cluster, and the storage node cluster has a cache and a data source, and the device includes: A receiving module, used to receive an identifier of a first scan operator and first data from a computing node cluster, wherein the first scan operator is generated based on a first data query command, the first data query instruction is received by the computing node cluster and is used to obtain a first query result from the data source, the data source is located in the storage node cluster, the first scan operator describes an operation required to be performed to obtain the first data from a first data object in the data source, the first data is used to obtain the first query result, and the first data corresponding to the identifier of the first scan operator is not stored in the cache of the storage node cluster; A storage module is used to store the identifier of the first scanning operator and the first data in the cache.
33. The device according to claim 32, characterized in that The computing node cluster and the storage node cluster are deployed separately.
34. The device according to claim 32 or 33, characterized in that The storage module is used to store the first data in a first directory, where the directory name of the first directory includes an identifier of the first scanning operator.
35. The device according to claim 34, characterized in that The first directory corresponds to first metadata, which includes metadata of the first directory, and the metadata of the first directory includes information of the first scanning operator, the size of the content contained in the first directory and / or the number of times the first directory is accessed.
36. The device according to claim 35, characterized in that When the information of the first directory is changed, the identifier of the first metadata is a first identifier; After the information of the first directory is changed, the identifier of the first metadata is a second identifier, and the second identifier is different from the first identifier.
37. A computing node cluster, characterized in that: The computing node cluster includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method described in any one of claims 1-15.
38. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 27 is implemented.
39. A storage node cluster, characterized in that: The storage node cluster includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method described in any one of claims 16-27.
Citation Information
Patent Citations
Dimension context propagation techniques for optimizing SQL query plans
CN111971666A
Data query method and device
CN112463814A
Method for injecting time series data, method for querying time series data and database system
CN113868267A
Semantic caching of semi-join operators in shared-nothing and log-structured databases
US20220245150A1
System and method for distributed query processing using configuration-independent query plans
US7984043B1
Cited By
Database retrieval method and device, equipment, storage medium and product
CN121456193A
A database retrieval method, apparatus, device, storage medium, and product.
CN121456193B
