Data processing method, system and apparatus, device and storage medium

By introducing acceleration nodes between computing and storage nodes to receive and process query commands, thereby reducing data transfer volume, the problem of low computational efficiency in big data processing is solved, and data transfer and computing performance are improved.

WO2024239867A1PCT designated stage expired Publication Date: 2024-11-28CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/088677
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-04-18
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In big data processing, the efficiency of data transmission and computation between computing nodes and storage nodes is limited by network bandwidth, resulting in low efficiency when reading large amounts of data and performing calculations.

Method used

An acceleration node is introduced between the compute node and the storage node. It receives query commands, obtains target data, performs computation processing, and then sends the computation results, thereby reducing the amount of data transmitted to the compute node and reducing the resource pressure on the compute node.

Benefits of technology

It improved data transmission efficiency, reduced the resource usage pressure on computing nodes, and increased the response speed of query services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088677_28112024_PF_FP_ABST
    Figure CN2024088677_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present application are a data processing method, system and apparatus, a device and a storage medium. The method is applied to an acceleration node located between a computation node and a storage node, and the method comprises: receiving a query instruction sent by a computation node, the query instruction comprising identification information of target data and a computation instruction needing to be executed for the target data; acquiring the target data according to the identification information of the target data, and processing the target data according to the computation instruction so as to obtain a computation result; and sending the computation result to the computation node. By means of the solution, the computation instruction originally needing to be executed on the computation node is pushed down to the acceleration node for execution, and the acceleration node only transmits the computation result to the computation node, thereby reducing the volume of data transmitted to the computation node, improving data transmission efficiency, and reducing the resource use pressure on the computation node.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing method, system, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310595005.2 and invention name “Data processing method, system, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of big data technology, and in particular to a data processing method, system, device, equipment and storage medium. Background Art

[0003] In the field of big data processing, a computing architecture that separates storage and computing is often used, with data computing and storage set on different nodes. Columnar storage formats are often used to store data, such as Parquet.

[0004] In practical applications, there is often a need to read large amounts of data and perform certain calculations (such as aggregation operators such as averaging and addition, and filtering operators such as numerical filtering and text screening). Due to the limitations of network bandwidth on the computing and storage sides, transmission efficiency and computing efficiency are often low.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a data processing method, system, apparatus, device, and storage medium for improving the data transmission efficiency and computing performance of computing nodes and storage nodes.

[0007] In a first aspect, an embodiment of the present application provides a data processing method, applied to an acceleration node located between a computing node and a storage node, the method comprising:

[0008] Receive a query instruction sent by a computing node, wherein the query instruction includes: identification information of target data and a computing instruction to be executed on the target data;

[0009] acquiring the target data according to the identification information of the target data;

[0010] Processing the target data according to the calculation instruction to obtain a calculation result;

[0011] Send the calculation result to the computing node.

[0012] In a second aspect, an embodiment of the present application provides a data processing system, the system comprising:

[0013] A computing node cluster, an acceleration node cluster, and a storage node cluster, wherein the acceleration node cluster is coupled between the computing node cluster and the storage node cluster;

[0014] Any acceleration node in the acceleration node cluster is used to execute the data processing method as described in the first aspect.

[0015] In a third aspect, an embodiment of the present application provides a data processing device, applied to an acceleration node located between a computing node and a storage node, the device comprising:

[0016] A receiving module, configured to receive a query instruction sent by a computing node, wherein the query instruction includes: identification information of target data and a computing instruction to be executed on the target data;

[0017] a processing module, configured to obtain the target data according to the identification information of the target data; and process the target data according to the calculation instruction to obtain a calculation result;

[0018] A sending module is used to send the calculation result to the computing node.

[0019] In a fourth aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data processing method described in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data processing method described in the first aspect.

[0021] In the solution provided in the embodiment of the present application, an acceleration node is set between the computing node and the storage node. When the computing node needs to query the target data from the storage node and perform corresponding calculations on the target data (such as filtering, aggregation, etc.), the acceleration node first receives the query instruction sent by the computing node, wherein the query instruction includes: identification information of the target data and the calculation instruction that needs to be executed on the target data; then, according to the identification information of the target data, the target data is obtained, and the target data is processed according to the calculation instruction to obtain the calculation result; finally, the calculation result is sent to the computing node. In this solution, the calculation instruction that originally needs to be executed on the computing node is pushed down to the acceleration node for execution, and the acceleration node only transmits the calculation result to the computing node. On the one hand, it can reduce the amount of data transmitted to the computing node and improve the data transmission efficiency; on the other hand, it can also reduce the resource usage pressure of the computing node and improve the response speed of the query service. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] FIG1 is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0024] FIG2 is a schematic diagram of the architecture of another data processing system provided in an embodiment of the present application;

[0025] FIG3 is a flow chart of a data processing method provided in an embodiment of the present application;

[0026] FIG4 is a flow chart of another data processing method provided in an embodiment of the present application;

[0027] FIG5 is a flowchart of another data processing method provided in an embodiment of the present application;

[0028] FIG6 is a schematic diagram of the structure of an acceleration node provided in an embodiment of the present application;

[0029] FIG7 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0030] FIG8 is a schematic structural diagram of an electronic device corresponding to the data processing device provided by the embodiment shown in FIG7 . DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0034] Figure 1 is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application. As shown in Figure 1, the data processing system includes: a computing node cluster, an acceleration node cluster, and a storage node cluster. The acceleration node cluster is coupled between the computing node cluster and the storage node cluster.

[0035] The computing node cluster contains N (N is greater than or equal to 1) different computing nodes. Each computing node is mainly deployed with computing resources such as CPU and certain storage resources to perform various computing tasks, such as data filtering and aggregation.

[0036] A storage node cluster consists of P (P greater than or equal to 1) different storage nodes. Each storage node primarily deploys storage resources such as disks and certain computing resources for persistent data storage. Data can be stored in either columnar or row-based formats.

[0037] The acceleration node cluster contains M (M is greater than or equal to 1) acceleration nodes. Each acceleration node has corresponding computing resources and storage resources. Based on the query instructions sent by the computing node, it performs data preprocessing on the target data queried by the computing node and feeds the obtained processing results with smaller data volumes back to the computing node to accelerate data transmission. The data preprocessing process will be explained in detail in the following embodiments. Here, we will focus on the structure of the data processing system.

[0038] The values ​​of N, P, and M can be the same or different. For ease of description, the following embodiments are described from the perspectives of computing nodes, storage nodes, and acceleration nodes, which are any nodes in the corresponding cluster.

[0039] It should be noted that in this embodiment, the acceleration node is located between the computing node and the storage node to emphasize that data queries between the computing node and the storage node must go through the acceleration node, rather than limiting the deployment of the hardware corresponding to the computing node, acceleration node and storage node.

[0040] During specific implementation, the acceleration node in this embodiment can be regarded as a software component, which can be deployed on a node with storage resources and computing resources, and its corresponding deployment conditions can be diverse.

[0041] For example, in a computing architecture where storage and computing are separated, acceleration nodes can be deployed on node clusters other than the computing node cluster and the storage node cluster, as shown in Figure 1, to achieve accelerated data transmission through the storage resources and computing resources on other node clusters.

[0042] If there is no other node cluster available for deploying the acceleration node, the acceleration node can also be deployed on the node cluster where the storage node is located, that is, the acceleration node and the storage node are deployed on the same node cluster, as shown in Figure 2, which is a schematic diagram of the architecture of another data processing system provided by an embodiment of the present application. In the scenario illustrated in Figure 2, the storage node and the acceleration node can be regarded as software components. For example, in actual applications, the storage node can be a distributed file system (Hadoop Distributed System, referred to as HDFS). The mixed deployment of the acceleration node and the storage node can not only make full use of the storage resources of the node, but also make full use of the computing resources of the node to provide acceleration services, thereby realizing accelerated data transmission between the storage node and the computing node.

[0043] Optionally, acceleration nodes can also be deployed on storage nodes, meaning that acceleration and storage are deployed on the same node. For example, for a storage node cluster containing P acceleration nodes, acceleration node 1 can be deployed on storage node 1, and acceleration node 2 can be deployed on storage node 2.

[0044] Regardless of which acceleration node is deployed, it can accelerate data transmission by reducing the amount of data transmitted to the compute nodes. For example, with the acceleration service provided by the acceleration node, the data query process between compute node 1 and storage node 1 changes from "compute node 1 → storage node 1 → compute node 1" (without acceleration node 1) to "compute node 1 → acceleration node 1 → storage node 1 → acceleration node 1 → compute node 1." Acceleration node 1 preprocesses the target data obtained from storage node 1, reducing the amount of data required to be transmitted to compute node 1.

[0045] In actual applications, different compute nodes may correspond to different computing frameworks. For example, compute node 1 corresponds to Spark, compute node 2 corresponds to Hadoop, and compute node 3 corresponds to Presto. Compute nodes can be divided into different types based on their corresponding computing frameworks. To ensure that different compute node types are compatible with acceleration nodes, data reading plug-ins corresponding to various compute node types are provided.

[0046] Optionally, the data processing system described in FIG1 further includes: a control node configured to determine, based on the type of the target compute node in the compute node cluster, a data reading plug-in that matches the type of the target compute node, and deploy the plug-in in the target compute node. Each acceleration node in the acceleration node cluster provides a unified service call interface that supports a specified data transmission protocol, wherein the specified data transmission protocol specifies the input and output data formats. The compute node calls the service call interface in the acceleration node through the data reading plug-in deployed therein, communicating with the acceleration node, such as issuing query instructions.

[0047] In actual applications, if there is no data reading plug-in that matches the computing node type, users can also develop a data reading plug-in that matches their own computing node type based on the unified service call interface in the acceleration node.

[0048] Optionally, a corresponding load balancing algorithm can be configured between the compute node cluster and the acceleration node cluster. Query instructions sent by a compute node can be sent to any acceleration node in the acceleration node cluster based on the load balancing algorithm. The communication between compute node 1 and acceleration node 1 in Figure 1 is merely illustrative; in actual applications, the communication between compute nodes and acceleration nodes is not fixed. For example, if acceleration node x is idle at time t1, query instruction 1 sent by the target compute node at time t1 will be sent to acceleration node x based on the load balancing algorithm. If acceleration node y is idle at time t2, query instruction 2 sent by the target compute node at time t2 will be sent to acceleration node y based on the load balancing algorithm, and so on.

[0049] The above embodiments only briefly introduce the basic architecture of the data processing system. The following describes the specific execution logic of the acceleration node when implementing the data processing method provided in the embodiments of the present application from the perspective of the acceleration node.

[0050] It should be noted that the data processing method of this embodiment is used to process data stored in a columnar format.

[0051] FIG3 is a flowchart of a data processing method provided in an embodiment of the present application, which is applied to an acceleration node located between a computing node and a storage node. As shown in FIG3 , the method may include the following steps:

[0052] 301. Receive a query instruction sent by a computing node, where the query instruction includes: identification information of target data and a computing instruction to be executed on the target data.

[0053] 302. Acquire target data according to identification information of the target data.

[0054] 303. Process the target data according to the calculation instruction to obtain a calculation result.

[0055] 304. Send the calculation result to the computing node.

[0056] In this embodiment, an acceleration node is any node in an acceleration node cluster, providing a unified service call interface that supports a specified data transmission protocol. Compute nodes are deployed with a data reading plug-in adapted for the compute node. This plug-in can call the service call interface in the acceleration node, enabling communication between the compute node and the acceleration node, such as receiving query commands from the compute node and sending computation results to the compute node.

[0057] The identification information of the target data in the query instruction may include storage path information of the target data, etc. The computation instructions to be executed on the target data in the query instruction may be understood as computation instructions corresponding to operators that would otherwise be executed on the target data on the compute node if no acceleration node existed.

[0058] Optionally, the computing instructions that need to be executed on the target data in the query instruction may be all or part of the computing instructions that originally need to be executed on the target data at the computing node.

[0059] For example, assuming the target data is data A, data B, and data C, and the compute node originally needs to average data A, data B, and data C, the corresponding computation instructions may include: Compute Instruction 1—Sum data A, data B, and data C to obtain computation result D; Compute Instruction 2—Split computation result D into three equal parts to obtain computation result E. Based on this assumption, the computation instruction that needs to be executed on the target data in the query instruction can be only Compute Instruction 1, or both Compute Instruction 1 and Compute Instruction 2.

[0060] In actual applications, the computing node may determine the computing instructions included in the query instruction that need to be executed on the target data based on the association relationship between the computing instructions executed on the target data.

[0061] Optionally, the computing node may also add the calculation instructions corresponding to the preset category of operators to the query instructions as the calculation instructions that need to be executed on the target data according to the type of operators that need to be executed on the target data. For example, the preset category of operators is simple operators such as scanning, filtering, aggregation, and field clipping. If the operator types that the computing node needs to execute on the target data include filtering and aggregation, the calculation instructions corresponding to the filtering and aggregation operators are added to the query instructions.

[0062] After obtaining the target data based on its identification information, the acceleration node performs the corresponding computation on the target data according to the computation instructions included in the query. Assuming the computation instruction included in the query is computation instruction 1, the acceleration node will sum data A, data B, and data C to obtain computation result D, and send computation result D to the computation node.

[0063] It is understandable that the large amount of target data obtained based on the target data's identification information can effectively reduce the amount of data after undergoing certain computational processing. For example, in the above example, if computation instruction 1 is not executed on the target data at the acceleration node, three pieces of data, namely data A, data B, and data C, need to be sent to the computing node. However, if computation instruction 1 is executed on the target data at the acceleration node, only one piece of data, namely the computation result D, needs to be sent to the computing node, effectively reducing the amount of data that needs to be transmitted. Furthermore, after receiving computation result D, the computing node only needs to execute computation instruction 2 to determine the average value of data A, data B, and data C, thereby effectively reducing the computational workload of the computing node and lowering the computing node's resource overhead, such as CPU and memory.

[0064] In this embodiment, the computing instructions that originally need to be executed on the computing node are pushed down to the acceleration node, so that the acceleration node pre-calculates the target data, that is, executes the computing instructions included in the query instruction that need to be executed on the target data, thereby effectively reducing the amount of data that needs to be transmitted to the computing node through the network, improving data transmission efficiency, and alleviating the computing pressure of the computing node.

[0065] FIG4 is a flowchart of another data processing method provided in an embodiment of the present application, which is applied to an acceleration node located between a computing node and a storage node. As shown in FIG4 , the method may include the following steps:

[0066] 401. Receive a query instruction sent by a computing node, where the query instruction includes: identification information of target data and a computing instruction to be executed on the target data.

[0067] 402. Based on the identification information of the target data, if it is determined that the acceleration node has the target data cached locally, the target data is obtained locally. If the acceleration node does not have the target data cached locally, the target data is retrieved from other acceleration nodes belonging to the same cluster as the acceleration node. If the acceleration node does not have the target data cached locally and the target data is not retrieved from other acceleration nodes belonging to the same cluster as the acceleration node, the target data is read from a storage node. The storage node is used to persistently store the target data.

[0068] 403. Process the target data according to the calculation instruction to obtain a calculation result.

[0069] 404. Send the calculation result to the computing node.

[0070] The specific implementation process of step 401, step 403 and step 404 can refer to the above embodiment and will not be described again here.

[0071] In this embodiment, the acceleration nodes have corresponding storage resources that can be used to cache target data. It is understood that retrieving target data from the local cache of an acceleration node is faster than retrieving it from a storage node. Furthermore, if multiple acceleration nodes are deployed simultaneously in the same acceleration node cluster, these multiple acceleration nodes can transfer locally cached target data to each other via the internal network. Furthermore, target data transmission between acceleration nodes in the same acceleration node cluster is faster than retrieving it from a storage node.

[0072] Therefore, to increase the speed at which the acceleration node acquires target data based on its data identifier, this embodiment sets corresponding judgment conditions when acquiring target data. This allows for rapid acquisition of target data. The example of a compute node communicating with acceleration node x in an acceleration node cluster illustrates the process of an acceleration node acquiring target data.

[0073] When acquiring target data, first, based on the identification information of the target data, it is determined whether the target data is cached in the local cache of acceleration node x. If the target data is cached, the target data is directly acquired from the local cache of acceleration node x.

[0074] If the target data does not exist in the local cache of acceleration node x, it then determines whether the target data is cached in other acceleration nodes in the same acceleration node cluster as acceleration node x. If the target data is cached in the local cache of another acceleration node, such as acceleration node y, the target data in acceleration node y is transferred to acceleration node x via the internal network between acceleration nodes y and x to obtain the target data. The target data in the local cache of acceleration node y may be data that was previously cached when acceleration node y provided acceleration services and has not yet been cleared.

[0075] If the target data is not found from other acceleration nodes in the same cluster as acceleration node x, the target data is finally read from the storage node used for persistent storage of the target data and stored in the local cache of acceleration node x.

[0076] It is understandable that the data cached locally by the acceleration node has a certain timeliness, and the data cached in the previous time period will be overwritten by the data cached in the next time period. When the target data has been loaded into the local cache of the acceleration node or loaded into the local cache of another acceleration node belonging to the same acceleration node cluster as the acceleration node before the acceleration node obtains the target data, and has not yet been overwritten, the target data acquisition method provided in the embodiment of the present application can effectively save the acquisition time of the target data and improve the acceleration performance of the acceleration node. In addition, the target data can be processed according to the calculation instructions to reduce the amount of data transmission between the local cache and the calculation node, thereby improving data transmission efficiency.

[0077] FIG5 is a flowchart of another data processing method provided in an embodiment of the present application, which is applied to an acceleration node located between a computing node and a storage node. The acceleration node includes a processor and a hardware accelerator. As shown in FIG5 , the method may include the following steps:

[0078] 501. Receive a query instruction sent by a computing node, where the query instruction includes: identification information of target data and a computing instruction to be executed on the target data.

[0079] 502. Acquire target data according to identification information of the target data.

[0080] 503. Determine whether to use a hardware accelerator to process the target data according to the data type of the target data and the instruction type of the calculation instruction; if so, process the target data according to the calculation instruction through the processor and the hardware accelerator to obtain a calculation result; if not, process the target data according to the calculation instruction through the processor to obtain a calculation result.

[0081] 504. Send the calculation result to the computing node.

[0082] The specific implementation process of step 501, step 502 and step 504 can refer to the above embodiment and will not be described again here.

[0083] In this embodiment, to improve the computing performance of the acceleration node's acceleration service, the acceleration node includes not only a processor (such as a CPU) that is general-purpose for multiple data types and computational instruction types, but also hardware accelerators such as FPGAs, GPUs, and AVX. The accelerated computing capabilities of the hardware accelerator are reflected in its superior computational processing capabilities for specific data types or computational instruction types. For example, FPGAs excel at string-type text matching and regular expression calculations, while GPUs excel at precise floating-point comparisons and operations. Compared to the processing efficiency of CPUs, FPGAs and GPUs offer higher processing efficiency for the data types or operator types they are best suited to.

[0084] In actual applications, multiple different hardware accelerators can be deployed simultaneously in the same acceleration node, such as FPGA, GPU, AVX, etc., to adapt to various data types and computing instruction types and provide better acceleration services.

[0085] Optionally, the data type of the target data may be included in the query instruction sent by the computing node, or may be determined based on the data type of the historical data.

[0086] During the specific implementation process, after obtaining the target data, the acceleration node will determine whether to use a hardware accelerator to process the target data based on the data type of the target data and the instruction type of the calculation instruction contained in the query instruction. In other words, it will determine whether there is a hardware accelerator in the acceleration node that matches the data type of the target data and the instruction type of the calculation instruction.

[0087] When there is no hardware accelerator in the acceleration node that matches the data type of the target data and the instruction type of the computing instruction, the hardware accelerator is not used to process the computing instruction on the target data. Instead, the processor processes the computing instruction on the target data to obtain the calculation result.

[0088] For ease of understanding, the following description is provided in conjunction with Figure 6. Figure 6 is a schematic diagram of the structure of an acceleration node provided in an embodiment of the present application. As shown in Figure 6, the acceleration node receives query instructions from the computing node through a processor and reads target data through the processor; the processor then processes the target data to obtain a calculation result and sends the calculation result to the computing node.

[0089] In actual applications, since storage nodes may be built by users themselves or purchased from cloud vendors, the data in the storage nodes is stored in columnar storage format, but the formats may be different. For example, storage node 1 stores target data in the columnar storage format of Parquet, while storage node 2 stores target data in the columnar storage format of ORC.

[0090] In this embodiment, after reading the target data, the processor may convert the target data's columnar storage format into a unified columnar storage format, also referred to as a first columnar storage format. The first columnar storage format may be the same as or different from the second columnar storage format corresponding to the target data in the storage node. After executing the computation instruction on the target data, the processor sends the computation result to the compute node, still in the first columnar storage format.

[0091] In this embodiment, the first columnar storage format can be Arrow. Due to Arrow's zero-copy nature, the processor can directly transmit computation results to and from compute nodes without serialization or deserialization. If the first columnar storage format is Arrow, the service call interface provided by the acceleration node supports the Arrow Flight protocol, which constrains input and output data formats to implement a unified service call interface.

[0092] When there is a hardware accelerator in the acceleration node that matches the data type of the target data and the instruction type of the computing instruction, the hardware accelerator is used to process the computing instruction on the target data. Specifically, the computing instruction is processed on the target data by the processor and the hardware accelerator to obtain the calculation result.

[0093] As shown in Figure 6, when the target data is processed by the processor and the hardware accelerator for the calculation instruction, first, the processor stores the read target data in the first column storage format to the first storage address in the memory of the acceleration node. Then, the first storage address and the calculation instruction contained in the query instruction that needs to be executed on the target data are sent to the hardware accelerator, so that the hardware accelerator obtains the target data according to the first storage address and processes the target data according to the calculation instruction to obtain the calculation result. Then, the hardware accelerator stores the calculation result in the first column storage format to the second storage address in the memory, and sends the second storage address to the processor. Finally, the processor obtains the calculation result from the second storage address and sends the calculation result to the computing node in the first column storage format.

[0094] In this embodiment, hardware accelerators with accelerated computing capabilities are deployed in acceleration nodes, enabling them to rapidly process data types and computational instructions that match the hardware accelerators, thereby improving the acceleration node's computational processing performance. Because a unified columnar storage format, such as Arrow, is used to transmit data between the acceleration node's processor and the hardware accelerator, and between the processor and the compute node, data format conversion processes are effectively reduced, resulting in efficient data transmission.

[0095] The data processing devices of one or more embodiments of the present application will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0096] Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application. The device is applied to an acceleration node located between a computing node and a storage node. As shown in Figure 7, the device includes: a receiving module 11, a processing module 12, and a sending module 13.

[0097] The receiving module 11 is configured to receive a query instruction sent by a computing node, wherein the query instruction includes identification information of target data and a computing instruction to be executed on the target data.

[0098] The processing module 12 is configured to obtain the target data according to the identification information of the target data; and process the target data according to the calculation instruction to obtain a calculation result.

[0099] The sending module 13 is configured to send the calculation result to the computing node.

[0100] Optionally, the acceleration node provides a unified service call interface, which supports a set data transmission protocol; the computing node is deployed with a data reading plug-in adapted to the computing node, so as to call the service call interface through the data reading plug-in and issue the query instruction.

[0101] Optionally, the acceleration node includes a processor and a hardware accelerator with accelerated computing capabilities. The processing module 12 is specifically configured to determine, based on the data type of the target data and the instruction type of the computing instruction, whether to use the hardware accelerator to process the target data according to the computing instruction; if so, perform the computing instruction processing on the target data using the processor and the hardware accelerator to obtain a computing result; if not, perform the computing instruction processing on the target data using the processor to obtain a computing result.

[0102] Optionally, the processing module 12 is further specifically configured to store the read target data in a first column storage format to a first storage address in the memory of the acceleration node, and the target data is read by the processor; send the first storage address and the calculation instruction to the hardware accelerator, so that the hardware accelerator obtains the target data according to the first storage address, processes the target data according to the calculation instruction to obtain a calculation result, stores the calculation result in the first column storage format to a second storage address in the memory, and sends the second storage address to the processor; and obtains the calculation result from the second storage address through the processor.

[0103] Optionally, the processing module 12 is further specifically configured to, based on the identification information of the target data, obtain the target data locally if it is determined that the acceleration node has the target data cached locally; if the acceleration node does not have the target data cached locally, and the target data is queried from other acceleration nodes belonging to the same cluster as the acceleration node, obtain the target data from the other acceleration nodes; if the acceleration node does not have the target data cached locally, and the target data is not queried from other acceleration nodes belonging to the same cluster as the acceleration node, read the target data from the storage node, and the storage node is configured to persistently store the target data.

[0104] Optionally, the target data is stored in the storage node in a second column storage format.

[0105] The device shown in FIG7 can execute the steps introduced in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0106] In one possible design, the structure of the data processing device shown in FIG. 7 can be implemented as an electronic device, as shown in FIG. 8 . The electronic device may include a memory 21, a processor 22, and a communication interface 23. The memory 21 stores executable code. When the executable code is executed by the processor 22, the processor 22 can at least implement the data processing method provided in the aforementioned embodiments.

[0107] In addition, an embodiment of the present application provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data processing method provided in the aforementioned embodiment.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0109] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, It is characterized in that Applied to an acceleration node located between a computing node and a storage node, the method includes: Receiving a query instruction sent by a computing node, the query instruction including: identification information of target data and a computing instruction to be executed on the target data; Acquire the target data according to the identification information of the target data; Processing the target data according to the calculation instruction to obtain a calculation result; The calculation result is sent to the computing node.

2. The method according to claim 1, It is characterized in that The acceleration node provides a unified service calling interface, and the service calling interface supports a set data transmission protocol; the computing node is deployed with a data reading plug-in adapted to the computing node, so as to call the service calling interface through the data reading plug-in and issue the query instruction.

3. The method according to claim 1, It is characterized in that The acceleration node includes a processor and a hardware accelerator; the processing of the target data according to the calculation instruction to obtain a calculation result includes: Determining whether to use the hardware accelerator to process the target data with the computing instruction according to the data type of the target data and the instruction type of the computing instruction; If yes, processing the calculation instruction on the target data by the processor and the hardware accelerator to obtain a calculation result; If not, the processor processes the target data according to the calculation instruction to obtain a calculation result.

4. The method according to claim 3, It is characterized in that The processing of the calculation instruction on the target data by the processor and the hardware accelerator to obtain a calculation result includes: storing the read target data in a first column storage format to a first storage address in the memory of the acceleration node, wherein the target data is read by the processor; Sending the first storage address and the calculation instruction to the hardware accelerator, so that the hardware accelerator obtains the target data according to the first storage address, processes the target data according to the calculation instruction to obtain a calculation result, stores the calculation result in the first column storage format to a second storage address in the memory, and sends the second storage address to the processor; The calculation result is obtained from the second storage address by the processor.

5. The method according to any one of claims 1 to 4, It is characterized in that The acquiring the target data according to the identification information of the target data includes: If it is determined that the acceleration node has the target data cached locally according to the identification information of the target data, then the target data is obtained locally; If the acceleration node does not cache the target data locally, the target data is queried from other acceleration nodes belonging to the same cluster as the acceleration node, and the target data is obtained from the other acceleration nodes; If the acceleration node does not cache the target data locally, If the target data is not found in the acceleration node, the target data is read from the storage node, and the storage node is used to persistently store the target data.

6. The method according to any one of claims 1 to 4, It is characterized in that The target data is stored in the storage node in a second column storage format.

7. A data processing system, It is characterized in that The system comprises: A computing node cluster, an acceleration node cluster and a storage node cluster, wherein the acceleration node cluster is coupled between the computing node cluster and the storage node cluster; Any acceleration node in the acceleration node cluster is used to execute the data processing method according to any one of claims 1 to 6.

8. The system according to claim 7, It is characterized in that The system further includes: a control node, which is used to determine, according to the type of the target computing node in the computing node cluster, a data reading plug-in that matches the type of the target computing node, so as to deploy the plug-in in the target computing node.

9. A data processing device, It is characterized in that Applied to an acceleration node located between a computing node and a storage node, the device comprises: A receiving module, configured to receive a query instruction sent by a computing node, wherein the query instruction includes: identification information of target data and a computing instruction to be executed on the target data; A processing module, configured to obtain the target data according to the identification information of the target data; and process the target data according to the calculation instruction to obtain a calculation result; A sending module is used to send the calculation result to the calculation node.

10. An electronic device, It is characterized in that include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the data processing method as described in any one of claims 1 to 6.

11. A non-transitory machine-readable storage medium, It is characterized in that The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the data processing method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Data processing system, method and apparatus

    CN108241722A

  • Data processing method and system and related equipment

    CN115509734A

  • Data processing method and device based on big data platform and storage medium

    CN115729442A

  • Data processing method, computing device and computer system

    CN115933975A

  • Data processing method, system, device and equipment and storage medium

    CN116680071A