STORAGE SYSTEM, DATA ACCESS METHOD, APPARATUS AND DEVICE

Abstract

A storage system, a data access method, an apparatus and a device are provided. When determining that target data needs to be accessed, the computing node first judges whether the target data meets the computation pushdown condition, and after determining that the target data meets the computation pushdown condition, it may send a data access request to the storage node. The data access request is used to access the target data, the data access request carries a computation instruction, and the computation instruction indicates the computation service that needs to be performed to access the target data. After receiving the data access request, the storage node obtains the dataset to which the target data belongs, executes the computation service according to the computation instruction, accesses the target data in the dataset, and feeds back the access result to the computing node. The computing node can determine a scenario suitable for computation pushdown, so that the resources of the storage node can be fully used and the effectiveness of computation pushdown can be ensured.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to Chinese Patent Application No. 202111585803.4, entitled "STORAGE SYSTEM, DATA ACCESS METHOD, APPARATUS, AND DEVICE," filed with the China State Intellectual Property Office on December 20, 2021, the entire contents of which are incorporated herein by reference.

[0002] [Technical field] The present application relates to the field of storage technology, and in particular to storage systems, data access methods, apparatus and devices. [Background technology]

[0003] A large object (LOB) is a data amount exceeding a gigabyte (GB), and a large object can be a large amount of data such as a document, a graphic image, a video or an audio. Currently, to meet the needs of users, all popular relational databases such as Oracle, MySQL, and PostgreSQL support large objects. In other words, data of the large object type can be stored in the database.

[0004] However, in a storage-computation separated storage system (i.e., the storage system includes computation nodes configured to implement computation functions and storage nodes configured to store data), the amount of large object type data is large, so when a storage node needs to transmit large object type data to a computation node, a large network bandwidth is occupied and a large transmission overhead occurs. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention provides a storage system, a data access method, an apparatus and a device for fully utilizing the resources of a storage node and efficiently implementing computation pushdown. [Means for solving the problem]

[0006] According to a first aspect, an embodiment of the present application provides a storage system. The storage system includes a computing node and a first storage node. The storage system is a storage-computation separated storage system. In the storage system, when determining that the target data needs to be accessed, the computing node first determines whether the target data meets the computation push-down condition, and after determining that the target data meets the computation push-down condition, the computing node may send a data access request to the first storage node. The data access request is used to access the target data, the data access request carries a computation instruction, and the computation instruction indicates a computation service that needs to be performed to access the target data. After receiving the data access request, the first storage node is configured to obtain a dataset to which the target data belongs, perform a computation service according to the computation instruction, access the target data in the dataset, and feed back the access result to the computing node.

[0007] According to the above system, the computing node can determine a scenario suitable for computation pushdown, so that the resources of the storage node can be fully utilized and the effectiveness of computation pushdown can be ensured.

[0008] In a possible implementation, the computation pushdown conditions include some or all of the following:

[0009] Condition 1: The data type is the specified data type.

[0010] Condition 2: The amount of data is greater than the data amount threshold.

[0011] Condition 3: The storage space that needs to be occupied by the data set to which the data belongs and the free storage space in the memory of the computing node satisfy a preset relationship.

[0012] Condition 4: The load of the storage node where the data is located is less than a load threshold.

[0013] Condition 5: The calculation method that needs to be performed on the data is a pre-defined calculation method.

[0014] Condition 6: The data access method does not support locking queries.

[0015] The above operation pushdown condition is merely an example. The specific content of the operation pushdown condition is not limited in this embodiment of the present application. Any condition that needs to be satisfied and is applicable to the scenario of implementing operation pushdown may be used as the operation pushdown condition.

[0016] According to the above system, there are multiple computation pushdown conditions that can be applied to different scenarios of computation pushdown.

[0017] In a possible implementation, the dataset to which the target data belongs may be distributed to different storage nodes, for example, distributed to a first storage node and a second storage node. In other words, a portion of the data in the dataset is stored in the first storage node and the second storage node. When retrieving the dataset to which the target data belongs, the first storage node may first determine the storage nodes to which the dataset is distributed, and retrieve the data in the dataset stored in the distributed storage nodes from the distributed storage nodes. For example, when determining that a portion of the data in the dataset is stored in the second storage node, the first storage node may retrieve the portion of the data from the second storage node.

[0018] According to the above system, in the storage system, the storage nodes can interact with each other to perform data transmission, which can avoid the need for the computing node to interact with the storage node where the data set is distributed multiple times, and can ensure that the computing pushdown can be performed efficiently and effectively.

[0019] In a possible implementation, when the first storage node executes a computation service based on the computation instruction to access the target data in the dataset, there are multiple computation services executed based on the computation instruction. Some types of computation services are listed below.

[0020] First type: Line clipping

[0021] The first storage node may extract one or more data rows from the dataset that satisfy a first condition, where the one or more data rows are target data.

[0022] Second type: Column clipping

[0023] The first storage node may extract one or more data strings from the target data that satisfy a second condition, where the one or more data strings are the target data.

[0024] Third type: Aggregation operations

[0025] After extracting the target data in the dataset, the first storage node may further perform an operation on the target data, such as adding, averaging, taking the minimum value, or taking the maximum value. In other words, it is necessary to further perform an operation service on the target data to obtain an operation result.

[0026] According to the above system, a storage node can execute multiple different computation services, which is applicable to different scenarios of computation pushdown and effectively expands the application scope.

[0027] According to a second aspect, an embodiment of the present application provides a data access method. The method can be executed by a computing node in a storage system. For advantageous effects, please refer to the relevant description of the first aspect. Details will not be described again here. In the method, when target data needs to be accessed, the computing node can first determine whether the target data meets the computation push-down condition. When determining that the target data to be accessed meets the computation push-down condition, the computing node sends a data access request to a storage node in a storage node, where the data access request is used to access the target data, the data access request carries a computation instruction, and the computation instruction indicates the computation service that needs to be performed to access the target data; then, the computing node can obtain the access result of the target data accessed by the storage node from the storage node.

[0028] In a possible implementation, the computation pushdown conditions include some or all of the following:

[0029] Condition 1: The data type is the specified data type.

[0030] Condition 2: The amount of data is greater than the data amount threshold.

[0031] Condition 3: The storage space that needs to be occupied by the data set to which the data belongs and the free storage space in the memory of the computing node satisfy a preset relationship.

[0032] Condition 4: The load of the storage node where the data is located is less than a load threshold.

[0033] Condition 5: The calculation method that needs to be performed on the data is a pre-defined calculation method.

[0034] Condition 6: The data access method does not support locking queries.

[0035] The above operation pushdown condition is merely an example. The specific content of the operation pushdown condition is not limited in this embodiment of the present application. Any condition that needs to be satisfied and is applicable to the scenario of implementing operation pushdown can be used as the operation pushdown condition and is applicable to this embodiment of the present application.

[0036] According to a third aspect, an embodiment of the present application provides a data access method. The method can be performed by a storage node in a storage system. For advantageous effects, please refer to the relevant description of the first aspect. Details will not be described again here. In the method, a storage node can receive a data access request from a computing node in a storage system, where the data access request is used to access target data, the data access request carries an operation instruction, and the operation instruction indicates a computing service that needs to be performed to access the target data. After receiving the data access request, the storage node can obtain a dataset to which the target data belongs, and perform a computing service according to the operation instruction to access the target data in the dataset, and can feed back the access result of accessing the target data to the computing node.

[0037] In a possible implementation, when a storage node retrieves a dataset to which the target data belongs, if the dataset is distributed to storage nodes and further distributed to another storage node, the storage node may retrieve data in the dataset stored locally and further retrieve data in the dataset from the other storage node. In other words, after determining that a portion of the data in the dataset is stored in another storage node in the storage system, the storage node may retrieve the portion of the data from the other storage node.

[0038] In a possible implementation, when a storage node executes a computation service based on a computation instruction to access target data in a dataset, there are multiple computation services that are executed based on the computation instruction. Some types of computation services are listed below.

[0039] First type: Line clipping

[0040] The storage node may extract one or more data rows from the dataset that satisfy a first condition, where the one or more data rows are target data.

[0041] Second type: Column clipping

[0042] The storage node may extract one or more data strings that satisfy a second condition from the target data, where the one or more data strings are the target data.

[0043] Third type: Aggregation operations

[0044] After extracting the target data in the dataset, the storage node may further perform an operation on the target data (such as adding, averaging, taking the minimum value, or taking the maximum value). In other words, the storage node needs to further perform an operation service on the target data to obtain an operation result.

[0045] According to a fourth aspect, an embodiment of the present application further provides a data access device. For advantageous effects, please refer to the description of the first aspect. Details will not be described again here. The device has a function of implementing the behavior in the method example of the second aspect. This function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the device includes a determining unit, a sending unit, and a receiving unit. These units may perform corresponding functions in the method example of the second aspect. For details, please refer to the detailed description in the method example. Details will not be described again here.

[0046] According to the fifth aspect, the embodiment of the present application further provides a data processing device. For advantageous effects, please refer to the description of the first aspect. Details will not be described again here. The device has a function of implementing the behavior in the method example of the third aspect. This function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the device includes a receiving unit, a computing unit, and a sending unit. These units may perform corresponding functions in the method example of the third aspect. For details, please refer to the detailed description in the method example. Details will not be described again here.

[0047] According to a sixth aspect, an embodiment of the present application provides a computing device. The computing device includes a processor and a memory, and may further include a network interface card. The processor may call program instructions in the memory to enable the computing device to execute a method according to the second aspect or any one of the possible implementations of the second aspect. The memory is coupled to the processor and stores program instructions and data necessary to execute the data transmission method. The network interface card is configured to communicate with another device to transmit data, for example, to send a data access request to the storage node and to receive an access result from the storage node.

[0048] Alternatively, the processor may call program instructions in the memory to enable the computing device to execute the method according to the third aspect or any one of the possible implementations of the third aspect. The memory is coupled to the processor and stores program instructions and data necessary to execute the data transmission method. The network interface card is configured to communicate with another device to transmit data, such as receiving a data access request from a receiving node and feeding back the access result to the computing node.

[0049] According to a seventh aspect, the present application provides a non-transitory readable storage medium. When the non-transitory readable storage medium is executed by a computing device, the computing device may execute the method according to the second aspect or any one of the possible implementations of the second aspect. Also, when the non-transitory readable storage medium is executed by a computing device, the computing device may alternatively execute the method according to the third aspect or any one of the possible implementations of the third aspect. The storage medium stores a program. The storage medium includes, but is not limited to, a volatile memory, such as a random access memory, and a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD).

[0050] According to an eighth aspect, the present application provides a computing device program product. The computing device program product includes computer instructions. When the computer instructions are executed by the computing device, the computing device may execute the method according to the second aspect or any one of the possible implementations of the second aspect. When the computer instructions are executed by the computing device, the computing device may alternatively execute the method according to the third aspect or any one of the possible implementations of the third aspect. The computer program product may be a software installation package. When it is necessary to use the method according to the second aspect or any one of the possible implementations of the second aspect, the computer program product may be downloaded and the computer program product may be executed on the computing device. [Brief description of the drawings]

[0051] [Figure 1] FIG. 1 is a diagram of the architecture of a system according to the present application.

[0052] [Diagram 2]FIG. 2 is a diagram of a data access method according to the present application.

[0053] [Diagram 3] FIG. 2 is a diagram of the structure of a data access device according to the present application;

[0054] [Figure 4] 1 is a diagram of the structure of a data computing device according to the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] In addition to the storage function, the storage system must also have a computation function. The computation function means that the storage system can perform simple processing on data, such as filtering and sorting. When a node in a storage system has both a storage function and a computation function, the storage system may be called a combined storage-computation storage system. When a storage system includes a node having a storage function and a node having a computation function, and the storage function and the computation function in the storage system are separated, the storage system may be called a separated storage-computation storage system.

[0056] FIG. 1 illustrates a storage system to which the embodiment is applicable. The storage system is a storage-computation separated storage system. The storage system includes a computation node cluster and a storage node cluster. The computation node cluster includes one or more computation nodes 110 (FIG. 1 illustrates two computation nodes 110, but is not limited to two computation nodes 110). The computation nodes 110 may communicate with each other. The computation nodes 110 are computing devices having computation capabilities. The computation nodes 110 may be servers, desktop computers, controllers of storage arrays, etc.

[0057] Functionally, the main function of the computation node 110 is to execute computation services. The computation node 110 may process data access instructions sent by clients. The computation node 110 may analyze the data access instructions initiated by the client to determine the target data that needs to be accessed by the client and the dataset to which the target data belongs, and may determine the storage node in which the dataset is located.

[0058] Using a database as an example, in the case of a relational database, the dataset may be understood as a table and the target data may be understood as a part of the data in the table, for example, as a part of a column or as a part of a row. In the case of a non-relational database, the dataset may be understood as a file containing multiple key-value pairs, a file containing multiple columns, or a graph containing multiple entities or entity relationships. The target data may be understood as a part of the key-value pairs or a part of a column in the file, or as a part of the entities and entity relationships in the graph.

[0059] In general, after the computing node 110 determines the dataset to which the target data belongs, the computing node 110 may initiate a data request to the storage node in which the dataset is located to request to obtain the dataset. After obtaining the dataset, the target data in the dataset is accessed based on the data access instruction to execute the computation service. However, in an embodiment of the present application, the computing node 110 has a judgment logic, and based on the judgment result, judges whether to push down part or all of the computation service to the storage node 100, and instructs the storage node 100 to access the data based on the computation instruction. Thus, in an embodiment of the present application, after determining the storage node in which the dataset is located, the computing node 110 may further determine whether the target data meets the computation pushdown condition. When the computing node 110 determines that the target data meets the computation pushdown condition, the computing node 110 may initiate a data access request to the storage node 100. The data access request is used to request access to the target data, and carries a computation instruction, where the computation instruction indicates the computation service that needs to be executed by the storage node 100.

[0060] 1, the computing node 110 includes a controller 111. The controller 111 includes at least a processor 112 and a memory 113. The computing node 110 may further include a network interface card 114.

[0061] Processor 112 may be a central processing unit (CPU) and is configured to process data access requests from outside of compute node 110 or requests generated within compute node 110 .

[0062] The memory 113 is an internal memory that directly exchanges data with the processor 112 and may be a random access memory or a read-only memory (ROM). The random access memory may be a dynamic random access memory (DRAM) or a storage class memory (SCM). The memory may further include another random access memory, for example a static random access memory (SRAM).

[0063] The network interface card 114 is configured to communicate with the storage node 100. The computing node 110 may send a request to the storage node 100 via the network interface card 114 to perform persistent storage on data, or may send a request to the storage node 100 via the network interface card 114 to retrieve data from the storage node. In addition, the computing node 110 may further include a bus used for communication between components within the computing node 110. During actual implementation, the computing node 110 may alternatively have a small number of internal hard disk drives or may be connected to a small number of external hard disk drives.

[0064] The storage node cluster includes multiple storage nodes 100 (although FIG. 1 shows three storage nodes 100, it is not limited to three storage nodes 100, and storage node 100A, storage node 100B and storage node 100C may be used to distinguish between the three storage nodes 100).

[0065] Functionally, the main function of the storage node 100 is to store data. Each storage node includes a physical storage space, which is the actual location where data is stored. The data stored in the storage node may be data in a database. Databases can be classified into relational and non-relational databases. In the case of a relational database, the storage node 100 may store a table in the relational database. In the case of a non-relational database for storage of key-value pairs, the storage node 100 may store key-value pairs in the non-relational database. In the case of a non-relational database for document storage, the storage node 100 may store a file (a file may contain multiple key-value pairs) in the non-relational database. In the case of a non-relational database for a column-oriented database, the storage node 100 may store a file or a table that contains multiple data columns and is in the non-relational database. In the case of a graph-based non-relational database, the storage node 100 may store a graph that includes one or more entities and entity relationships and is within the non-relational database.

[0066] In general, the storage node 100 only needs to feed back the dataset to which the target data belongs to the computing node 110 based on the data request of the computing node 110. In an embodiment of the present application, after receiving the data access request of the computing node 110, the storage node 100 may obtain the dataset to which the target data belongs. The storage node 100 may further access the target data in the dataset based on the data access request, and perform the computing service.

[0067] Structurally, one storage node 100 includes one or more control units 101, a network interface card 104, and a number of hard disk drives 105. The network interface card 104 is configured to communicate with a computing node 110. The hard disk drive 105 is configured to store data and may be a magnetic disk or another type of storage medium, such as a solid-state drive or a single magnetic recording hard disk drive. The control unit 101 is configured to write data to or read data from the hard disk drive 105 based on a data access request sent by the computing node 110. In the process of reading or writing data, the control unit 101 needs to convert an address carried in the data access request into an address that can be recognized by the hard disk drive.

[0068] The control unit 101 may have various forms. For example, the control unit 101 may include a CPU and a memory. The CPU is configured to perform operations such as address translation and data reading or writing. The memory is configured to temporarily store data to be written to the hard disk drive 105 or to read data from the hard disk drive 105 to be sent to the computation node 110. In another case, the control unit 101 may alternatively be a programmable electronic component, such as a data processing unit (DPU). Optionally, the DPU may alternatively be replaced by a processing chip such as a graphics processing unit (GPU) or an embedded neural-network processing unit (NPU).

[0069] As another example, the functions of the control unit 101 may be offloaded to the network interface card 104. In other words, the storage node 100 does not have the control unit 101, and the network interface card 104 performs data reading and writing, address conversion, and other computing functions. In this case, the network interface card 104 is an intelligent network interface card. The memory 103 is configured to temporarily store data to be written to the hard disk drive 105, or to read data from the hard disk drive 105 to be sent to the computing node 110. There is no homing relationship between the network interface card 104 and the hard disk drive 105 in the storage node 100, and the network interface card 104 may access any hard disk drive 105 in the storage node 100. Therefore, it is convenient to expand the hard disk drive when the storage space is insufficient.

[0070] In an embodiment of the present application, in order to effectively use the resources (e.g., the control unit 101) of the storage node 100, the computing node 110 may push down some or all of the computing services in the computing node 110 to the storage node 100, in other words, some computing services that need to be implemented in the computing node 110 may be assigned to the storage node 100. The computation pushdown can be implemented in a storage-computation separated storage system. In an embodiment of the present application, the computing node 110 has a decision logic and can determine a scenario for implementing the computation pushdown. This can effectively use the resources of the storage node 100 and improve the computation pushdown efficiency to a certain extent.

[0071] Hereinafter, the data access method provided in the embodiment of the present application will be described with reference to Figure 2. The method includes the following steps:

[0072] Step 201: The computation node 110 determines that target data currently needs to be accessed.

[0073] Scenario 1: A user may initiate, via a client deployed at the user side, a data access instruction to a computation node 110, which is used to access target data. The data access instruction may carry the logical address of the target data or the logical address of the dataset to which the target data belongs. Optionally, the data access instruction may further include an identifier of the dataset to which the target data belongs, etc.

[0074] There are many ways to access target data, which are indicated by data access instructions. For example, if a user needs to retrieve a portion of the data in a table (a portion of the rows or a portion of the columns of the data), the data access instructions instruct to retrieve a portion of the data in the table. In another example, if a user needs to retrieve the amount of rows that meet certain requirements and are in the table, the data access instructions instruct to calculate the amount of rows that meet certain requirements and are in the table.

[0075] Scenario 2: When data in storage node 100 needs to be processed (eg, to perform processing such as error detection or clustering on data in storage node 100), computation node 110 determines the target data that needs to be accessed.

[0076] Step 202: The computation node 110 determines whether the target data satisfies the computation pushdown condition.

[0077] Upon determining that the target data needs to be accessed, the computation node 110 may determine whether the target data satisfies a computation pushdown condition. In this embodiment of the present application, the computation pushdown condition includes some or all of the following:

[0078] the data type is a specified data type, the amount of data is greater than a data amount threshold, the storage space that needs to be occupied by the dataset to which the data belongs is greater than the free storage space in the memory of the computing node 110, the load of the storage node 100 in which the data is located is less than a load threshold, the computation method that needs to be performed on the data is a pre-configured computation method, and the data access method does not support lock queries.

[0079] The six conditions are explained separately below.

[0080] Condition 1: The data type is the specified data type.

[0081] In order to store the data in the database, the storage node 100 also stores some auxiliary data, such as temporary data, metadata and system data used to describe the storage system.

[0082] Auxiliary data is typically data that needs to be analyzed by the computation node 110 and does not need to be calculated by the storage node 100. When the target data is auxiliary data, computation pushdown is not applicable. When the target data is data in a database, computation pushdown can be performed.

[0083] Condition 2: The amount of data is greater than the data amount threshold.

[0084] When computation pushdown is not performed, when reading data from storage node 100, computation node 110 obtains the data set to which the data belongs from storage node 100. After obtaining the data set, computation node 110 either extracts the data it needs to access from the data set or processes the data set to generate a processing result.

[0085] In the case of a relational database, the dataset to which the target data belongs can be a table. In the case of a non-relational database, the dataset to which the target data belongs can be a file or a graph. Using a relational database as an example. Usually, the target data that needs to be accessed by the computing node 110 is a part of the data in a table. When acquiring data from the storage node 100, the computing node 110 acquires the entire table to which the target data belongs. After acquiring the table, the computing node 110 can perform a computing service on the table. For example, the computing node 110 may acquire the data that needs to be accessed from the table, or may access the table to acquire the access result. In this embodiment of the present application, the computing node 110 does not need to acquire the entire table, and the storage node 100 extracts data from the table or processes the table. In other words, the computing node 110 can assign a computing service to the storage node 100.

[0086] When the computation node 110 allocates a computation service to the storage node 100, the computation node 110 may directly indicate the target data that the storage node 100 needs to access, and the storage node 100 feeds back to the computation node 110 the results of accessing the target data.

[0087] When a storage node 100 executes a computational service, it knows that it needs to process a data set, the amount of data being typically reduced through such processing.

[0088] For a data set with a large amount of data, when the storage node 100 executes the computation service to process the data set, the amount of data reduced is large, the computation push-down effect is obvious, and the performance is greatly improved. For a data set with a small amount of data, when the storage node 100 executes the computation service to process the data set, the amount of data reduced is small, the computation push-down effect is weak, and the performance is slightly improved. That is, the larger the data set to which the data accessed by the computation node 110 belongs, the more obvious the computation push-down effect is.

[0089] Thus, the computation node 110 may configure a data volume threshold. When the data volume of the dataset to which the data belongs is greater than the data volume threshold, computation pushdown is applicable. When the data volume of the dataset to which the data belongs is not greater than the data volume threshold, computation pushdown is not applicable.

[0090] Condition 3: The storage space occupied by the data set to which the data belongs and the free storage space in the memory of the computing node satisfy a preset relationship. For example, the storage space required to be occupied by the data set to which the data belongs is equal to or greater than the free storage space in the memory of the computing node 110. In another example, the difference between the storage space required to be occupied by the data set to which the data belongs and the free storage space in the memory of the computing node 110 is greater than a threshold value.

[0091] When computation pushdown is not performed, after the computation node 110 acquires a dataset from the storage node 100, the acquired dataset may be stored in the memory of the computation node 110. If the storage space that needs to be occupied by the dataset and the free storage space on the memory of the computation node 110 are close and satisfy a preset relationship (e.g., one of the storage space that needs to be occupied by the dataset and the free storage space is equal to or larger than the other, and the difference between the storage space that needs to be occupied by the dataset and the free storage space is greater than a threshold), the dataset may not be completely present on the memory of the computation node 110.

[0092] From the above description, it can be seen that when the storage node 100 executes a computation service, the amount of data is usually reduced. In this case, after the storage node 100 processes the data set, the storage space that needs to be occupied by the processed data is probably equal to or less than the free storage space in the memory of the computation node 110. In this case, computation pushdown is applicable.

[0093] Condition 4: The load of the storage node 100 in which the data is located is less than a load threshold.

[0094] The computation pushdown is implemented to effectively use the resources (e.g., the control unit 101) of the storage node 100. In other words, the prerequisite for the computation pushdown is that there is an idle resource in the storage node 100 that can be used to perform computation services. The load of the storage node 100 may represent the currently occupied resources of the storage node 100. When the load of the storage node 100 is high, it is not preferable for the storage node 100 to perform computation services. When the load of the storage node 100 is low, there is an idle resource in the storage node 100 that can be used to perform computation services. Therefore, the computation node 110 may set a load threshold of the storage node 100. When determining whether it is necessary to perform computation pushdown, the computation node 110 may interact with the storage node 100 where the data is located to determine the current load of the storage node 100. The storage node 100 where the data is located may be determined by the computation node 110 based on the logical address of the data.

[0095] Condition 5: The calculation method that needs to be performed on the data is a pre-defined calculation method.

[0096] In computation pushdown, not all computation services are suitable to be pushed down to storage node 100 for execution by storage node 100. Computation node 110 may selectively select computation services suitable for pushing down to storage node 100. A computation service may be understood as a computation method performed on data.

[0097] In this embodiment of the application, the computation node 110 may configure one or more computation methods suitable for execution by the storage node 100. In the case of a relational database, the one or more computation methods suitable for execution by the storage node 100 may be to extract data at a row granularity, or to extract data at a column granularity, or to aggregate data. In the case of a non-relational database, the one or more computation methods suitable for execution by the storage node 100 may be to extract a particular key-value pair, or to extract data in a particular column, or to aggregate a particular key-value pair or column, or to extract a portion of an entity or entity relationship in a graph.

[0098] Condition 6: The data access method does not support locking queries.

[0099] A lock query means that the target data that needs to be queried cannot be modified, in which case the computation node usually does not need to perform computation services on the discovered target data, and accordingly, the computation node does not need to allocate computation services to the storage nodes.

[0100] For example, a data access request initiated by a client indicates that the data that needs to be read is used for update or sharing (i.e., select for update / share). The data needs to be used for update or sharing, in other words, the data that needs to be read should be the original data, i.e., locking queries are not supported. In this case, computation pushdown is not applicable. Another example is a query operation with serializable isolation level.

[0101] Unlike locking queries, there are query methods that support modification. For example, a data access request initiated by a client may indicate the reading of a row of data from a table named user, i.e., Select count(*) from user.

[0102] Step 203: After determining that the target data meets the computation pushdown condition, the computation node 110 sends a data access request to the storage node 100 where the target data is located, where the data access request is used to request access to the target data, and the data access request carries a computation instruction. Here, an example is used in which the storage node 100 where the target data is located is the storage node 100A.

[0103] The computation node 110 determines in which storage node 100 the target data is located based on the logical address of the target data or the logical address of the data set to which the target data belongs. After determining in which storage node 100A the data is located, the computation node 110 may send a data access request to storage node 100A.

[0104] If the computation node 110 determines that the target data meets the computation pushdown condition, this indicates that the computation service can be assigned to the storage node 100. The computation node 110 may include a computation instruction in the data access request. The computation instruction indicates the computation method that needs to be performed when accessing the target data. In other words, the computation instruction is used to inform the storage node 100A of the computation service that needs to be performed.

[0105] Step 204: After receiving the data access request, the storage node 100A accesses the target data according to the computation instruction.

[0106] When accessing the target data based on the operation instruction, the storage node 100A may first obtain the dataset to which the target data belongs, and then access the target data in the dataset based on the operation instruction.

[0107] When receiving a data access request from the computing node 110, the storage node 100A further needs to perform a layer-by-layer conversion on the logical address of the data set to convert the logical address into a physical address that can be identified by the hard disk drive. After obtaining the physical address, the storage node 100 can obtain the data set to which the target data belongs from the hard disk drive based on the physical address.

[0108] In a storage node cluster, data is stored at the smallest granularity. For convenience of explanation, the smallest unit of data storage is called a page, and the page size may be fixed. The storage space of each storage node 100 may be divided into one or more pages. For small data, the data may be stored as a whole in one page. For large data, such as LOBs, the data needs to be stored in multiple pages, and the multiple pages may be distributed to different storage nodes 100. In other words, when a large amount of data is distributed to different storage nodes 100, in a storage node cluster, the storage nodes 100 jointly maintain global paging layout information, which records the page in which the data is located and the storage node 100 in which the page is located.

[0109] When storage node 100A obtains the data set to which the target data belongs, it can be seen that there are two possible cases.

[0110] Case 1: The data set is stored in storage node 100A.

[0111] After storage node 100A receives a data access request initiated by computation node 110, storage node 100A determines, based on the logical address of the target data or the logical address of the dataset carried in the data access request and the global paging layout information, that the dataset to which the target data belongs is distributed to storage node 100A. Storage node 100A may retrieve the dataset based on the logical address of the dataset.

[0112] Case 2: The data set is distributed across multiple storage nodes 100, and storage node 100A is just one of the multiple storage nodes 100.

[0113] In general, when a data set is distributed to multiple different storage nodes 100, the storage node 100 where the data set is located, which is determined by the computation node 110 based on the logical address of the data set, is only one of the multiple different storage nodes 100. When the entire data set needs to be accessed, the computation node 110 initiates a data request to the determined storage node 100A to access the data set. After receiving the data request, the storage node 100 may determine, based on the global paging layout information, another storage node 100 where the data set is distributed and the address of the data set in the other storage node 100 (actually, this address is the address of a part of the data in the data set in the other storage node 100). The storage node 100 informs the computation node 110 of the other storage node 100 where the data is distributed and the address of the data set in the other storage node 100. In this way, the computation node 110 may send a data request to the other storage node 100 to request to obtain a part of the data in the data set from the other storage node 100. This data access method often occurs when the data being accessed is a large object, in which the compute node 110 interacts with the storage node 100 multiple times to retrieve the entire data set.

[0114] In this embodiment of the present application, in order to reduce the number of interactions between the computing node 110 and each storage node 100, reduce the amount of data transmitted between the computing node and the storage node, and ensure that the storage node 100 can more appropriately allocate computing services, after the storage node 100A receives a data access request initiated by the computing node 110, if the storage node 100A determines, based on the logical address of the target data or the logical address of the dataset carried in the data access request and the global paging layout information, that the dataset to which the target data belongs is distributed across multiple storage nodes 100 classified into storage node 100A, storage node 100C, and storage node 100B, in addition to obtaining a portion of the data in the dataset stored locally, the storage node 100A may interact with storage node 100C and storage node 100B to obtain the remaining data in the dataset from storage node 100C and storage node 100B.

[0115] Regardless of which method is used, after obtaining the data set, the storage node 100A may access the target data in the data set based on the computation instruction and perform the computation service.

[0116] In the case of a relational database, three calculation methods are listed below.

[0117] Method 1: Line Clipping

[0118] Row clipping involves clipping a table in a database at the granularity of rows and extracting only a portion of the rows that meet a requirement.

[0119] Using a MySQL database as an example, a table named customer contains columns such as json_column and gender, where json_column is of JSON data type, and the following expression may be used as an operation directive: json_column->"$.id" >= 1 AND json_column->"$.id" <= 10. This expression indicates that rows with json_column identifier (id) greater than or equal to 1 and less than or equal to 10 may be selected from the table named customer.

[0120] Method 2: Column Clipping

[0121] Row clipping involves clipping a table in a database at the granularity of columns and extracting only a portion of the columns that meet a requirement.

[0122] Continuing with the above table named customer as an example, the calculation directive may dictate that only two columns are to be extracted, json_column and gender, and no other columns need to be extracted.

[0123] Method 3: Aggregation operations

[0124] An aggregation operation is performing summarization on tables in a database to obtain a summary result, including but not limited to counting, taking the maximum value, taking the minimum value, averaging, sum, etc.

[0125] Using count as an example, the expression SELECT COUNT(json_column) FROM customer may be used as a calculation instruction to indicate that you want to count the json_column column of a table named customer.

[0126] Optionally, when a table named customer is distributed across different storage nodes 100, the computation node 110 may indicate one of the storage nodes 100 to count the table named customer, where the storage node 100 is to count a portion of the data in the stored table named customer, and may further indicate a storage node 100 other than the storage node 100 to count a portion of the data of the stored table named customer. Another storage node 100 may feed back a count value to the storage node 100. After the count value obtained by the storage node 100 through counting and the count value fed back by the other storage node 100 are summarized, the summarized count value is fed back to the computation node 110.

[0127] Alternatively, the computation node 110 may instruct each storage node 100 to count a portion of the data in a table named customer stored in the storage node 100, and obtain the count values ​​from the storage nodes 100. The computation node 110 then summarizes the obtained count values.

[0128] For non-relational databases, three calculation methods are listed below.

[0129] Method 1: Attribute extraction

[0130] The method is applicable to non-relational databases for storing key-value pairs or for storing files. Attribute extraction refers to extracting only a part of the key-value pairs.

[0131] Method 2: Column Clipping

[0132] The method is applicable to non-relational databases for column-oriented storage. Column clipping refers to the ability to extract only a portion of the columns in a file.

[0133] Method 3: Aggregation operations

[0134] The aggregation operation in a non-relational database is similar to that in a relational database. For details, please refer to the above explanation. The details will not be described again here.

[0135] Method 4: Entity Extraction

[0136] This method can be used for graph-based non-relational databases. Entity extraction indicates that only some of the entities or some of the entity relationships in the graph can be extracted.

[0137] The above calculation method is merely an example. The calculation method performed by the calculation instruction is not limited in this embodiment of the present application. Any calculation method that can be performed by the storage node 100 is applicable to this embodiment of the present application.

[0138] Step 205: The storage node 100A feeds back the access result to the operation node 110.

[0139] After accessing the target data according to the operation instruction, the storage node 100A obtains an access result. The access result may be the target data extracted from the data set, or may be a result obtained by performing an aggregation operation on the target data. The storage node 100 may feed back the access result to the operation node 110.

[0140] Based on the same inventive concept as the method embodiment, the embodiment of the present application further provides a data access apparatus, which is configured to execute the method executed by the computing node 110 in the method embodiment shown in Fig. 2. As shown in Fig. 3, the data access apparatus 300 includes a determining unit 301, a sending unit 302 and a receiving unit 303. These units may be software modules. Specifically, in the data access apparatus 300, connections between the units are established via communication paths.

[0141] The determining unit 301 is configured to determine that the target data to be accessed satisfies a computation pushdown condition.

[0142] The sending unit 302 is configured to send a data access request to a storage node within the storage node, where the data access request is used to access target data, the data access request carries an operation instruction, and the operation instruction indicates a operation service that needs to be performed to access the target data.

[0143] The receiving unit 303 is configured to obtain, from the storage node, an access result of the access by the storage node to the target data.

[0144] In a possible implementation, the computation pushdown conditions include some or all of the following:

[0145] The data type is the specified data type.

[0146] The amount of data is greater than the data amount threshold.

[0147] A predetermined relationship is satisfied between the storage space that needs to be occupied by the data set to which the data belongs and the free storage space in the memory of the computing node.

[0148] The load of the storage node where the data is located is below a load threshold.

[0149] The calculation method that needs to be performed on the data is a pre-defined calculation method.

[0150] The data access method does not support locking queries.

[0151] Based on the same inventive concept as the method embodiment, the embodiment of the present application further provides a data processing device, which is configured to execute the method performed by the storage node 100A in the method embodiment shown in Fig. 2. As shown in Fig. 4, the data processing device 400 includes a receiving unit 401, a computing unit 402, and a sending unit 403. These units may be software modules. Specifically, in the data processing device 400, connections between the units are established via communication paths.

[0152] The receiving unit 401 is configured to receive a data access request from a computing node in the storage system, where the data access request is used to access target data, the data access request carries a computing instruction, and the computing instruction indicates a computing service that needs to be performed to access the target data.

[0153] The computing unit 402 is configured to obtain a dataset to which the target data belongs, perform a computing service based on a computing instruction, and access the target data in the dataset.

[0154] The sending unit 403 is configured to feed back the access result of accessing the target data to the computing node.

[0155] In a possible implementation, when the computing unit 402 retrieves the dataset to which the target data belongs, if the computing unit 402 determines that some data in the dataset is stored in another storage node in the storage system, the computing unit 402 may retrieve the some data from the other storage node.

[0156] In a possible implementation, when the computing unit 402 executes a computing service based on the computing instruction and accesses the target data in the data set, there are multiple computing services that are executed based on the computing instruction. Some types of computing services are listed below.

[0157] First type: Line clipping

[0158] The computation unit 402 may extract one or more rows of data from the dataset that satisfy a first condition, where the one or more rows of data are target data.

[0159] Second type: Column clipping

[0160] The computation unit 402 may extract one or more data strings that satisfy a second condition from the target data, where the one or more data strings are the target data.

[0161] Third type: Aggregation operations

[0162] After extracting the target data in the dataset, the calculation unit 402 may further perform a calculation on the target data, such as adding, averaging, taking the minimum value, or taking the maximum value. In other words, it is necessary to further perform a calculation service on the target data to obtain a calculation result.

[0163] In the embodiment of the present application, the division into units is an example and is merely a logical function division. Other division methods may be used during actual implementation. In addition, the functional units in the embodiment of the present application may be integrated into one processor, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0164] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be implemented in the form of a software product, essentially, or the part that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium and includes some instructions for instructing a terminal device (which may be a personal computer, a mobile phone, a network device, etc.) or a processor to execute all or part of the steps of the method in the embodiments of the present application. The above-mentioned storage medium includes various media that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0165] The above-mentioned description of the procedures corresponding to the accompanying drawings has a focus on each. For the parts of the procedures that are not described in detail, please refer to the relevant description of another procedure.

[0166] All or part of the above-described embodiments may be implemented by software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product. The data synchronization computer program product includes one or more data synchronization computer program instructions. When the data synchronization computer program instructions are loaded into a computer and executed, all or part of the data synchronization procedures or functions according to the embodiments of the present application are generated.

[0167] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or another programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (e.g., coaxial cable, optical fiber, or digital subscriber line) or wireless (e.g., infrared, radio, or microwave) manner. The computer-readable storage medium includes a readable storage medium that stores computer program instructions for training a classification model. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device such as a server or data center that incorporates one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk drive, or a magnetic tape), an optical medium (e.g., a DVD), or a solid-state medium (e.g., an SSD).

Claims

1. A storage system comprising a computing node and a first storage node, The computing node is configured to send a data access request to the first storage node after determining that the target data to be accessed satisfies a computation pushdown condition, the data access request being used to access the target data, the data access request carrying a computation instruction, the computation instruction indicating a computation service that needs to be executed to access the target data; the first storage node is configured to receive the data access request, obtain a dataset to which the target data belongs, execute the computation service according to the computation instruction, access the target data in the dataset, and feed back an access result to the computation node; Storage system.

2. The computation pushdown conditions may be any or all of the following: The data type is the specified data type, and The amount of data is greater than a data amount threshold; and a storage area that needs to be occupied by the data set to which the data belongs and a free storage area in the memory of the computing node satisfy a preset relationship; the load of the storage node in which the data is located is less than a load threshold; the calculation method that needs to be performed on the data is a preset calculation method; The data access method does not support locking queries, and The system of claim 1 , comprising some or all of the following:

3. The storage system further includes a second storage node, and when acquiring the data set to which the target data belongs, the first storage node specifically: determining that a portion of the data in the data set is stored in the second storage node; obtaining a portion of the data from the second storage node; The system according to claim 1 or 2, configured to:

4. When executing the computation service based on the computation instruction and accessing the target data in the data set, the first storage node specifically: configured to extract from the dataset one or more rows of data that satisfy a first condition, the one or more rows of data being the target data; A system according to any one of claims 1 to 3.

5. When executing the computation service based on the computation instruction and accessing the target data in the data set, the first storage node specifically: configured to extract one or more data strings that satisfy a second condition from the target data, the one or more data strings being the target data; A system according to any one of claims 1 to 3.

6. When executing the computation service based on the computation instruction and accessing the target data in the data set, the first storage node specifically: configured to extract the target data from the dataset and execute the computation service on the target data to obtain a computation result. A system according to any one of claims 1 to 3.

7. 1. A data access method, the method being executed by a computing node in a storage system, the method comprising: determining that the target data to be accessed satisfies a computation pushdown condition; sending a data access request to a storage node within a storage node, the data access request being used to access the target data, the data access request carrying a computation instruction, the computation instruction indicating a computation service that needs to be performed to access the target data; obtaining, from the storage node, an access result of the access to the target data by the storage node; A data access method including:

8. The computation pushdown conditions may be any or all of the following: The data type is the specified data type, and The amount of data is greater than a data amount threshold; and a storage area that needs to be occupied by the data set to which the data belongs and a free storage area in the memory of the computing node satisfy a preset relationship; the load of the storage node in which the data is located is less than a load threshold; the calculation method that needs to be performed on the data is a preset calculation method; The data access method does not support locking queries, and The method of claim 7, comprising any or all of the following:

9. 1. A data access method, the method being executed by a storage node in a storage system, the method comprising: receiving a data access request from a compute node in the storage system, the data access request being used to access target data, the data access request carrying a computational instruction, the computational instruction indicating a computational service that needs to be performed to access the target data; obtaining a dataset to which the target data belongs, and executing the computation service based on the computation instruction to access the target data in the dataset; feeding back a result of accessing the target data to the operation node; A data access method including:

10. The step of acquiring a data set to which the target data belongs includes: determining that a portion of the data in the data set is stored in another storage node in the storage system; obtaining a portion of the data from the other storage node; 10. The method of claim 9, comprising:

11. Executing, by the storage node, the computation service based on the computation instruction and accessing the target data in the dataset, comprises: extracting from the dataset one or more rows of data that satisfy a first condition, the one or more rows of data being the target data; 11. The method according to claim 9 or 10.

12. Executing, by the storage node, the computation service based on the computation instruction and accessing the target data in the dataset, comprises: extracting one or more data strings that satisfy a second condition from the target data, the one or more data strings being the target data; 11. The method according to claim 9 or 10.

13. Executing, by the storage node, the computation service based on the computation instruction and accessing the target data in the dataset, comprises: extracting the target data from the dataset and executing the computation service on the target data to obtain a computation result; 11. The method according to claim 9 or 10.

14. A data access apparatus, the apparatus comprising: a determining unit, a sending unit, and a receiving unit; The determining unit is configured to determine that the target data to be accessed satisfies a computation pushdown condition; The sending unit is configured to send a data access request to a storage node in a storage node, the data access request being used to access the target data, the data access request carrying a computation instruction, the computation instruction indicating a computation service that needs to be performed to access the target data; The receiving unit is configured to obtain, from the storage node, an access result of the storage node accessing the target data; Data access device.

15. The computation pushdown conditions may be any or all of the following: The data type is the specified data type, and The amount of data is greater than a data amount threshold; and a storage area that needs to be occupied by the data set to which the data belongs and a free storage area in the memory of the computing node satisfy a preset relationship; the load of the storage node in which the data is located is less than a load threshold; the calculation method that needs to be performed on the data is a preset calculation method; The data access method does not support locking queries, and 15. The apparatus of claim 14, comprising some or all of the following:

16. A data processing device, the device comprising: a receiving unit, a processing unit, and a sending unit; The receiving unit is configured to receive a data access request from a computing node in a storage system, the data access request being used to access target data, the data access request carrying a computing instruction, the computing instruction indicating a computing service that needs to be performed to access the target data; The computing unit is configured to obtain a dataset to which the target data belongs, perform the computing service based on the computing instruction, and access the target data in the dataset; The sending unit is configured to feed back an access result of accessing the target data to the computing node. Data calculation device.

17. When obtaining the data set to which the target data belongs, the computing unit specifically: determining that a portion of the data of the data set is stored in another storage node within the storage system; configured to obtain a portion of the data from the other storage node; 17. The apparatus of claim 16.

18. When executing the computation service according to the computation instruction and accessing the target data in the data set, the computation unit specifically: configured to extract from the dataset one or more rows of data that satisfy a first condition, the one or more rows of data being the target data; 18. Apparatus according to claim 16 or 17.

19. When executing the computation service according to the computation instruction and accessing the target data in the data set, the computation unit specifically: configured to extract one or more data strings that satisfy a second condition from the target data, the one or more data strings being the target data; 18. Apparatus according to claim 16 or 17.

20. When executing the computation service according to the computation instruction and accessing the target data in the data set, the computation unit specifically: configured to extract the target data from the dataset and execute the computation service on the target data to obtain a computation result.

18. Apparatus according to claim 16 or 17.

21. A computing device, the computing device comprising a processor and a memory; the memory is configured to store computer program instructions; The processor invokes the computer program instructions in the memory to perform the method of any one of claims 7 to 13. Computing device.

22. A non-transitory computer readable storage medium, which when executed by a computing device, causes the computing device to perform the method of any one of claims 7 to 13.

Images