Data processing method, device, equipment and system of fusion system

Abstract

The invention discloses a data processing method, device, equipment and system of a fusion system, and relates to the field of data. The fusion system comprises a computing node and a storage node. The computing nodes are connected with the storage nodes through a network to construct a storage and computing separation architecture. And the storage medium of the computing node and the storage medium of the storage node are uniformly addressed to form a global memory pool, namely a global storage medium shared by the computing node and the storage node. When read-write operation is carried out on the system, processing request data is obtained, and memory operation of processing the request data is carried out on the global memory pool according to the memory operation instruction. Therefore, on the basis of the storage and calculation separation architecture with the global memory pool, calculation and storage are respectively and flexibly expanded according to needs, read-write operation can be quickly carried out on the system, the end-to-end data processing time is shortened, the data transmission speed is improved, the system energy consumption is reduced, and the system performance bottleneck is improved.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on April 8, 2022, with application number 202210369243.7 and application name “A Fusion System of Computing and Storage”, all contents of which are incorporated by reference in this application.

[0002] This application is a divisional application. The application number of the original application is 202210633694.7, and the original application date is June 6, 2022. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0003] The present application relates to the field of data, and in particular to a data processing method, device, equipment and system for a fusion system. Background Art

[0004] Storage and computing are two key systems for distributed applications (such as big data and databases), which determine the overall performance and energy consumption of the system. Usually, computing and storage are deployed on the same node, and the system performance is high, but the storage-computing ratio is fixed, and computing and storage cannot be flexibly expanded separately. Subsequently, the storage-computing separation architecture came into being. The storage cluster and the computing cluster are connected through the network, which enables flexible expansion of computing and storage on demand. However, data needs to go through multiple conversion operations such as protocols and formats from the computing side to the storage side, resulting in a long data processing time and high system energy consumption, and system performance has become a bottleneck. Summary of the invention

[0005] The present application provides a data processing method, device, equipment and system for a fusion system, thereby shortening data processing time, increasing data transmission speed and reducing system energy consumption.

[0006] In a first aspect, a data processing method for a fusion system is provided. The fusion system includes a computing node and a storage node. The computing node is connected to the storage node through a network to construct a storage-computing separation architecture. The storage medium of the computing node and the storage medium of the storage node are uniformly addressed to form a global memory pool, that is, a global memory shared by the computing node and the storage node. When performing a read and write operation on the system, processing request data is obtained, and a memory operation of the processing request data is performed on the global memory pool according to a memory operation instruction.

[0007] In this way, since memory operation instructions are a technology that uses memory interfaces to perform memory operations on memory, memory operations are performed on the global memory shared by computing nodes and storage nodes according to memory operation instructions to process memory operations of requested data, avoiding the need for data to go through multiple conversion operations such as protocols and formats from the computing side to the storage side, and effectively simplifying the data movement process between computing nodes and storage nodes. Therefore, the storage and computing separation architecture based on a global memory pool ensures that computing and storage can be flexibly expanded on demand, and can quickly read and write operations on the system, shortening the end-to-end data processing time, improving data transmission speed, reducing system energy consumption, and improving system performance bottlenecks.

[0008] Among them, the memory operation instruction includes at least one of memory allocation, memory setting, memory copying, memory moving, memory release and memory comparison. The embodiment of the present application does not limit the type of storage medium of the global memory pool. The storage medium of the global memory pool includes memory, hard disk, memory server and storage-class-memory (SCM). In this way, a global memory pool containing multiple storage media is introduced in a distributed processing system, so that the storage medium for storing data has more possibilities, and a matching storage medium is selected to store data, so that the system can be quickly read and written, which improves the data transmission speed and shortens the end-to-end data processing time.

[0009] In a possible implementation, performing a memory operation on a global memory pool to process request data according to a memory operation instruction includes: reading the data to be processed from the global memory pool; processing the data to be processed using the memory operation instruction according to the processing request data to obtain processed data, and writing the processed data into a storage space indicated by a first address in the global memory pool. The storage space indicated by the first address includes one of a storage space provided by a storage medium of a computing node and a storage space provided by a storage medium of a storage node.

[0010] The computing node may determine the storage space indicated by the first address according to the storage policy. For example, the computing node may determine the storage space indicated by the first address according to the access characteristics of the application.

[0011] Example 1: Determine, based on user needs and storage medium characteristics, to write the processed data to the storage space indicated by the first address in the global memory pool. Storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, CPU consumption, energy efficiency, and reliability. Thus, the system is read and written based on user needs and the system's storage medium characteristics, which not only enhances the user's control over the system, improves the user's system experience, but also expands the application scenarios applicable to the system.

[0012] Example 2: Determine, based on user needs and storage medium characteristics, that the processed data is written to the storage space indicated by the first address in the global memory pool, including: Determine, based on user needs and storage medium characteristics, that the processed data is written to the storage space of the computing node indicated by the first address in the global memory pool. Thus, the processed data is stored in the local memory of the computing side in the global memory pool for local memory access, which effectively shortens the data processing time and improves the data transmission speed.

[0013] Example 3, determining to write the processed data to the storage space indicated by the first address in the global memory pool according to user needs and storage medium characteristics, includes: determining to write the processed data to the storage space of the storage node indicated by the first address in the global memory pool according to user needs and storage medium characteristics. Thus, the processed data is stored in the memory on the storage side of the global memory pool, which improves the reliability and persistence of the data.

[0014] In another possible implementation, after writing the processed data into the storage space indicated by the first address in the global memory pool, the method further includes: reading the processed data from the global memory pool according to the first address.

[0015] For example, when the data stored in the global memory pool is saved to disk, the processed data is read from the global memory pool according to the first address, including: when the processed data needs to be persisted, the computing node reads the processed data from the global memory pool according to the first address and writes the processed data to the storage node.

[0016] For another example, when a computing node in the fusion system uses processed data, the computing node reads the processed data from the global memory pool according to the first address.

[0017] In another possible implementation, the method further includes: when reading processed data from the global memory pool according to the first address, performing another memory operation of processing the requested data on the global memory pool according to the memory operation instruction.

[0018] Therefore, from the perspective of application, the efficiency of data processing and the speed of data transmission are improved, and from the perspective of hardware equipment, the utilization rate of system resources is improved.

[0019] In another possible implementation, the method further includes: pre-fetching data from the storage node according to the memory operation instruction and storing the data in the global memory pool, so that the computing node can quickly obtain the data and shorten the data processing time.

[0020] In another possible implementation, the method further includes: performing memory operations on data between a global memory pool and a storage node based on memory operation instructions according to the hot and cold characteristics of the data. Cold data refers to data with a low access frequency. Cold data is moved from the global memory pool to the storage node, thereby releasing the storage space of the global memory pool and improving the utilization rate of the storage space of the global memory pool. Hot data refers to data with a high access frequency. Moving hot data from the storage node to the global memory pool enables the computing node to obtain the required data as quickly as possible, thereby shortening the data processing time and reducing the computing resources occupied by frequently reading and writing data.

[0021] In a second aspect, a data processing device for a fusion system is provided, the device comprising modules for executing the data processing method for the fusion system in the first aspect or any possible design of the first aspect.

[0022] In a third aspect, a computing device is provided, which includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions as a computing node in the first aspect or any possible implementation of the first aspect, the operating steps of the data processing method of the fusion system in the first aspect or any possible implementation of the first aspect are executed.

[0023] In a fourth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0024] In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0025] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of distributed data processing provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a deployment scenario of a global memory pool provided in an embodiment of the present application;

[0029] Figure 4A schematic diagram of a three-layer storage system provided in an embodiment of the present application;

[0030] Figure 5 A process diagram of a data processing method provided in an embodiment of the present application;

[0031] Figure 6 A process diagram of another data processing method provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of a process of Map task and Reduce task processing based on a global memory pool provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] With the development of services such as the Internet, the Internet of Things, network bandwidth, smart terminals, and cloud computing, the type and scale of data have developed at an unprecedented rate. Applications such as big data, databases, and high-performance computing (HPC) have emerged, and data has transformed from a single processing object to a basic resource.

[0036] Big data is a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain period of time. Since the large amount of data contained in big data is related to each other, data analysis methods, models, or tools are used to analyze big data, mine the data relationships in big data, and use the data relationships in big data to make predictions or decisions. For example, by analyzing user shopping trend data, users can be pushed items that they may buy to improve their shopping experience. Therefore, big data has the characteristics of large data volume, fast data growth rate, diverse data types, and high utilization value.

[0037] A database is a computer software system that stores and manages data in a data structure. It can also be understood that a database is a collection of data in a computer that is used to store and manage a large amount of data, that is, an electronic filing cabinet. Data can include travel records, consumption records, web pages browsed, messages sent, images, music, and sounds, etc.

[0038] Since the amount of data in these applications is very large, a single computing node cannot meet the computing needs. Usually, a distributed data processing system is used to process the data.

[0039] Figure 1The following is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application. Figure 1 As shown, data processing system 100 includes client 110 , computing cluster 120 , and storage cluster 130 .

[0040] The storage cluster 130 includes at least two storage nodes 131. A storage node 131 includes one or more controllers, a network card and multiple hard disks. The hard disk is used to store data. The hard disk can be a magnetic disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The network card is used to communicate with the computing node 121 included in the computing cluster 120. The controller is used to write data to the hard disk or read data from the hard disk according to the read / write data request sent by the computing node 121. In the process of reading and writing data, the controller needs to convert the address carried in the read / write data request into an address that the hard disk can recognize.

[0041] The client 110 communicates with the computing cluster 120 and the storage cluster 130 through the network 140. For example, the client 110 sends a service request to the computing cluster 120 through the network 140, requesting the computing cluster 120 to perform distributed processing on the service data included in the service request. The network 140 may refer to an internal network of an enterprise (such as a local area network (LAN)) or the Internet.

[0042] The data processing system 100 supports running applications such as big data, databases, high-performance computing, artificial intelligence, distributed storage, and cloud native. It is understandable that the business data in the embodiments of the present application includes data of applications such as big data, databases, high-performance computing, artificial intelligence (AI), distributed storage, and cloud native.

[0043] In some embodiments, the storage cluster 130 stores and manages a large amount of data based on the distributed file system 132 and the distributed database 133. The client 110 is installed with a client program 111. The client 110 runs the client program 111 to display a user interface (UI). The user 150 operates the user interface to access the distributed file system 132 and the distributed database 133 to obtain data and instruct the computing cluster 120 to process data services. The client 110 can refer to a computer connected to the network 140, which can also be called a workstation. Different clients can share resources on the network (such as computing resources and storage resources).

[0044] The computing cluster 120 includes at least two computing nodes 121, and the computing nodes 121 can communicate with each other. The computing node 121 is a computing device, such as a server, a desktop computer, or a controller of a storage array. For example, the big data business submitted by the client 110 can be called a job. The job can be divided into multiple tasks, and multiple computing nodes execute multiple tasks in parallel. The completion of all tasks marks the completion of a job. A task is generally a processing process for a part of data or a stage in a job, and all tasks are scheduled to be completed in parallel or serially. In some embodiments, the computing cluster 120 performs distributed processing on the big data business based on the map-reduce (MapReduce) model 134. The MapReduce model is a distributed programming model, which decomposes the big data business into mapping tasks and reduce tasks, and multiple computing nodes 121 execute map tasks, collect processing results and execute reduce tasks. In some embodiments, such as Figure 2 As shown, the computing cluster 120 includes a control node 122 and at least two computing nodes 121. The control node and the computing node can be independent physical devices, and the control node can also be called a control device or a naming node. The computing node can be called a computing device or a data node. The control node 122 is used to manage the namespace of the distributed file system and the access of the client 110 to the distributed file system. In addition, the control node 122 indicates the computing node that executes the map task and the computing node that executes the reduce task. The computing node executes the map task based on the MapReduce model 134 to obtain the intermediate data (or called map data or shuffle data), and stores the intermediate data in the global memory pool 170. The computing node that executes the reduce task reads the intermediate data from the global memory pool 170. Optionally, the storage cluster 130 can also process the data according to the MapReduce model 134 and then store it when storing data.

[0045] In other embodiments, the system administrator 160 can call the application platform interface (API) 112 or the command-line interface (CLI) interface 113 or the graphical user interface (GUI) through the client 110 to access the distributed file system 132 and the distributed database 133 to configure system information, such as the deployment information and storage strategy of the global memory pool 170 configured with unified addressing for the computing nodes 121 and the storage nodes 131 provided in the embodiments of the present application.

[0046] The embodiment of the present application provides a storage-computing fusion architecture and a data processing method based on memory operation instructions, that is, a global memory pool composed of the storage medium of the computing node and the storage medium of the storage node through unified addressing, and a technology for performing memory operations on the global memory pool to process request data according to memory operation instructions. This solves both the flexibility problem of separate expansion of storage and computing, and the performance problem of storage and computing separation. The memory operations of processing request data on the global memory pool according to memory operation instructions described in the embodiment of the present application include read operations and write operations on the global memory pool. A data processing system configured with a global memory pool may also be referred to as a fusion system. Memory operation instructions are a technology that uses memory interfaces to perform memory operations on storage media. Since the two different memory operations on the computing side and the storage side are integrated through a global memory pool that is globally uniformly compiled based on memory operation instructions, that is, memory operations are performed on the global storage medium shared by the computing nodes and the storage nodes to process requested data according to the memory operation instructions. This effectively simplifies the data movement process between the computing nodes and the storage nodes, and avoids the need for data to go through multiple conversion operations such as protocols and formats from the computing side to the storage side. Therefore, the storage and computing separation architecture based on the global memory pool ensures that computing and storage can be flexibly expanded as needed, and can quickly perform read and write operations on the system, shortening the end-to-end data processing time, improving data transmission speed, reducing system energy consumption, and improving system performance bottlenecks.

[0047] For example, format conversion includes serialization and deserialization. Serialization refers to the process of converting an object into a byte sequence. Deserialization refers to the process of restoring a byte sequence into an object.

[0048] It should be noted that the global memory pool provided in the embodiment of the present application may include the storage medium of the computing node and the storage medium of the storage node in the data processing system. The storage medium of the computing node includes at least one of the local storage medium in the computing node and the extended storage medium connected to the computing node. The storage medium of the storage node includes at least one of the local storage medium in the storage node and the extended storage medium connected to the storage node.

[0049] For example, the global memory pool includes local storage media within a computing node and local storage media within a storage node.

[0050] For another example, the global memory pool includes a local storage medium in a computing node, an extended storage medium connected to the computing node, and any one of a local storage medium in a storage node and an extended storage medium connected to the storage node.

[0051] For another example, the global memory pool includes local storage media in computing nodes, extended storage media connected to computing nodes, local storage media in storage nodes, and extended storage media connected to storage nodes.

[0052] For example, Figure 3 As shown, it is a schematic diagram of a deployment scenario of a global memory pool provided in an embodiment of the present application. The global memory pool 300 includes a storage medium 310 in each of the N computing nodes, an extended storage medium 320 connected to each of the N computing nodes, a storage medium 330 in each of the M storage nodes, and an extended storage medium 340 connected to each of the M storage nodes.

[0053] It should be understood that the storage capacity of the global memory pool may include part of the storage capacity in the storage medium of the computing node and part of the storage capacity in the storage medium of the storage node. The global memory pool is a storage medium that can be accessed by both computing nodes and storage nodes with unified addressing. The storage capacity of the global memory pool can be used by computing nodes or storage nodes through memory interfaces such as large memory, distributed data structure, data cache, metadata, etc. The computing node running the application can use these memory interfaces to perform memory operations on the global memory pool. In this way, the global memory pool constructed based on the storage capacity of the storage medium of the computing node and the storage medium of the storage node provides a unified memory interface for computing nodes to use, so that the computing node uses a unified memory interface to write data to the storage space provided by the computing node of the global memory pool or the storage space provided by the storage node, thereby realizing the calculation and storage of data based on memory operation instructions.

[0054] The above is an example of constructing a global memory pool using storage media in computing nodes and storage media in storage nodes. The deployment method of the global memory pool can be flexible and varied, and the embodiments of the present application are not limited thereto. For example, the global memory pool is constructed by the storage media of the storage nodes. For example, a high-performance storage layer (Global Cache). For another example, the global memory pool is constructed by the storage media of the computing nodes. For example, a high-performance storage layer. Using the storage media of a separate storage node or the storage media of a computing node to construct a global memory pool can reduce the occupancy of storage resources on the storage side, and provide a more flexible expansion solution.

[0055] It should be noted that memory is a memory device used to store programs and various data. The larger the memory capacity, the slower the access speed. Conversely, the smaller the memory capacity, the faster the access speed. Access speed refers to the data transmission speed when writing data to or reading data from the memory. Access speed can also be called read and write speed. Memories can be divided into different levels based on storage capacity and access speed.

[0056] For example, Figure 4 A schematic diagram of a three-layer storage system provided in an embodiment of the present application. From the first layer to the third layer, the storage capacity increases step by step, the access speed decreases step by step, and the cost decreases step by step. Figure 4As shown, the first level includes registers 411, a first-level cache 412, a second-level cache 413, and a third-level cache 414 located in a central processing unit (CPU). The memory included in the second level can be used as the main memory of the computer system. For example, dynamic random access memory (DRAM) 421, double data rate synchronous dynamic random access memory (DDR SDRAM) 422, storage-class-memory (SCM) 423. The main memory can be referred to as main memory or memory, that is, the memory that exchanges information with the CPU. The memory included in the third level can be used as auxiliary memory of the computer system. For example, network memory 431, solid-state drive (Solid State Disk or Solid State Drive, SSD) 432, hard disk drive (HDD) 433. Auxiliary memory can be referred to as auxiliary memory or external memory. Compared with the main memory, the external memory has a large storage capacity and a slow access speed. It can be seen that the closer the memory is to the CPU, the smaller the capacity, the faster the access speed, the larger the bandwidth, and the lower the latency. Therefore, the memory included in the third level stores data that the CPU does not frequently access, improving the reliability of the data. The memory included in the second level can be used as a cache device to store data that the CPU frequently accesses, significantly improving the access performance of the system.

[0057] The storage medium of the global memory pool provided in the embodiment of the present application includes memory (such as DRAM), SSD, hard disk, memory server and SCM.

[0058] In some embodiments, a global memory pool can be set according to the type of storage medium, that is, a memory pool is constructed using one type of storage medium, and different types of storage media are used to construct different types of global memory pools, so that the global memory pool is applied to different scenarios, and the computing node selects the storage medium according to the access characteristics of the application, which enhances the user's control over the system, improves the user's system experience and expands the application scenarios applicable to the system. For example, the DRAM in the computing node and the DRAM in the storage node are uniformly addressed to form a DRAM memory pool. The DRAM memory pool is used for application scenarios with high access performance requirements, moderate data capacity, and no data persistence requirements. For another example, the SCM in the computing node and the SCM in the storage node are uniformly addressed to form an SCM memory pool. The SCM memory pool is used for application scenarios that are not sensitive to access performance, have large data capacity, and have a demand for data persistence.

[0059] Different storage media have different storage medium characteristics. Storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, CPU consumption, energy consumption ratio and reliability. Write latency refers to the latency of computing node 121 writing data to the storage medium. Read latency refers to the latency of computing node 121 reading data from the storage medium. Storage capacity refers to the total storage capacity of the storage medium that can store data. Available storage capacity refers to the remaining storage capacity after deducting the used storage capacity from the total storage capacity. Access speed refers to the speed at which computing node 121 performs read and write operations on the storage medium. CPU consumption refers to the occupancy rate of the CPU of computing node 121 used by computing node 121 to write data to the storage medium or read data from the storage medium. Energy consumption ratio refers to the energy consumed per unit time (such as electrical energy). Reliability refers to the degree of persistence of data stored in the storage medium.

[0060] The implementation of the data processing method of the fusion system provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0061] Figure 5 A flowchart of a data processing method of a fusion system provided in an embodiment of the present application. Here, the client 110, the computing cluster 120 and the storage cluster 130 are used as examples for explanation. The computing cluster 120 includes a control node 122 and at least two computing nodes 121, and the storage cluster 130 includes at least two storage nodes 131. The control node 122 is used to control the computing node 121 to perform data distributed processing. The computing node 121 and the storage node 131 are configured with a global memory pool 170. The global memory pool 170 includes the storage medium of the computing node 121 and the storage medium of the storage node 131. Figure 5 As shown, the method includes the following steps.

[0062] Step 510: The computing node 121 obtains processing request data.

[0063] The client 110 responds to the user operation and sends a service request for the data service to the control node 122. The control node 122 can receive the service request for the big data service sent by the client 110 through the local area network or the Internet. The service request may include a service identifier and service data. The service identifier is used to uniquely indicate a data service. The service data may be data for data distributed processing by the computing node or identification data indicating data to be processed.

[0064] User operations can refer to operations in which users operate the big data user interface to submit big data services. Big data services include data analysis services, data query services, and data modification services. For example, big data services refer to analyzing customers' personal data and purchase behavior data to describe user portraits and classify customers, so that targeted products or preferential products can be recommended to specific customers, improving customer satisfaction, and consolidating customer relationships. Another example is that big data services refer to analyzing the historical sales volume of products to predict future sales volume, discovering the reasons for the decline or increase in sales volume, and recommending constructive suggestions to increase sales.

[0065] User operations may also refer to operations in which users operate the database user interface to submit database services, etc. Database operations include database creation, deletion, modification, and query, etc.

[0066] The computing node 121 can receive the service request sent by the control node 122, and convert the service request into processing request data that complies with the memory operation instruction operation rules, so that the computing node 121 can perform memory operations on the processing request data on the global memory pool according to the memory operation instruction.

[0067] Step 520: The computing node 121 performs a memory operation on the global memory pool to process the requested data according to the memory operation instruction.

[0068] Memory operation instructions may also be referred to as memory semantics or memory operation functions. Memory operation instructions include at least one of memory allocation (malloc), memory setting (memset), memory copy (memcpy), memory move (memmove), memory release (memoryrelease) and memory comparison (memcmp).

[0069] Memory allocation is used to allocate a section of memory to support application execution.

[0070] Memory settings are used to set the data mode of the global memory pool, such as initialization.

[0071] Memory copy is used to copy data stored in the storage space indicated by the source address (source) to the storage space indicated by the destination address (destination).

[0072] The memory move is used to copy the data stored in the storage space indicated by the source address (source) to the storage space indicated by the destination address (destination), and delete the data stored in the storage space indicated by the source address (source).

[0073] Memory comparison is used to compare whether the data stored in two storage spaces are equal.

[0074] Memory release is used to release data stored in the memory to improve the utilization of system memory resources and thus improve system performance.

[0075] The processing request data is used to indicate the operation to be performed on the data to be processed. For example, the processing request data indicates obtaining the sales volume of product A in the first quarter. The data to be processed may include the sales volume for the whole year. The specific operation process included in step 520 is described in the following steps 521 to 523.

[0076] Step 521 : The computing node 121 reads the data to be processed and the application data.

[0077] The data to be processed and the application data may be stored in a storage space such as a global memory pool, a storage medium of a storage node, or a storage medium of a computing node. The computing node 121 may read the data to be processed and the application data from the global memory pool, the storage medium of a storage node, or the storage medium of a computing node. For example, the computing node 121 reads the application data from the local memory and reads the data to be processed from the global memory pool. The data to be processed may be an object that needs to be processed according to the indication of the processing request data. The application data includes application programs and application configuration data.

[0078] Step 522: The computing node 121 starts the application program according to the application data, and processes the data to be processed using the memory operation instruction according to the processing request data to obtain processed data.

[0079] For example, assuming that the application data includes database application programs and database configuration data, and the processing request data is used to indicate sorting of terabyte (TB) of data to be processed, the computing node 121 starts the database according to the database application data, obtains the data to be processed, and sorts the data to be processed.

[0080] It should be noted that the computing node 121 may start the application program according to the application data before obtaining the processing request data, or may start the application program according to the application data after obtaining the processing request data.

[0081] In this way, data processing based on memory operation instructions can be applied to the entire life cycle of data processing. Using memory operation instructions for data processing can achieve de-input / output (IO) and ultra-fast end-to-end data access. Combined with the global memory pool of storage and computing fusion, format and protocol conversion during data movement can be avoided, effectively reducing the end-to-end data processing time.

[0082] Step 523: The computing node 121 writes the processed data into the storage space indicated by the first address in the global memory pool.

[0083] The computing node 121 automatically selects a storage medium for storing processed data from the global memory pool according to the storage policy. The storage policy includes access characteristics of the application and characteristics of the storage medium in the global memory pool. The storage space indicated by the first address includes one of the storage space provided by the storage medium of the computing node 121 and the storage space provided by the storage medium of the storage node 131.

[0084] In some embodiments, the computing node 121 determines to write the processed data into the storage space indicated by the first address in the global memory pool according to the user demand and the storage medium characteristics. The user demand is used to indicate the demand related to the storage medium characteristics. The processing request data includes the user demand. The storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, CPU consumption, energy consumption ratio and reliability.

[0085] For example, computing node 121 is configured with storage medium characteristics of multiple types of storage media. User demand indicates an access speed range or a specific access speed, and computing node 121 determines the storage medium that meets the user demand from the global memory pool. For example, user demand indicates the access speed of the memory. Computing node 121 selects a storage medium that meets the access speed of the memory from the global memory pool, such as at least one of memory, DRAM, and SCM. Example 1: Computing node 121 determines to write the processed data to the storage space of the computing node indicated by the first address in the global memory pool based on user demand and storage medium characteristics, and the storage space of the computing node indicated by the first address meets the access speed indicated by the user demand. The processed data is thereby stored in the local memory on the computing side of the global memory pool for local memory access, which effectively shortens the data processing time and improves the data transmission speed.

[0086] For another example, computing node 121 is configured with an association relationship between storage media and customer levels. User demand indicates a first customer level. Computing node 121 determines the storage medium associated with the first customer level from the association relationship according to the first customer level, and determines that the storage medium associated with the first customer level is used to store processed data. Example 2: Computing node 121 determines to write the processed data into the storage space of the storage node indicated by the first address in the global memory pool according to the user demand and the storage medium characteristics, and the storage space of the computing node indicated by the first address meets the first customer level indicated by the user demand.

[0087] In this way, when computing node 121 selects a storage medium for storing processed data from the global memory pool, it dynamically selects a storage medium that meets user requirements based on user requirements for storage medium characteristics such as access speed or reliability, thereby ensuring that data processing performance and reliability meet scenario requirements.

[0088] In other embodiments, the computing node 121 is configured with priorities of multiple types of storage media determined according to storage medium characteristics, and the storage medium used to store processed data is determined according to the priorities of multiple types of storage media indicated by the storage policy. Among them, the priorities of multiple types of storage media can be determined based on the access speed of the storage medium. For example, the access speed of the memory is higher than the access speed of the hard disk, and the access speed of the hard disk is higher than the access speed of the extended storage medium. The priorities of multiple types of storage media can be determined based on the priorities of the deployment mode. For example, the priority of the local storage medium is higher than the priority of the extended storage medium. The priorities of multiple types of storage media can be determined comprehensively based on the characteristics of multiple types of storage media. The priorities of multiple types of storage media are determined based on the priorities of the deployment mode (such as: local storage medium and extended storage medium). The priority of storage media under the same deployment mode can be determined based on the access speed of the storage medium.

[0089] Optionally, the computing node 121 may also select a storage medium with an available storage capacity greater than a threshold from a plurality of types of storage media that meet user requirements as the storage medium for storing the processed data.

[0090] The computing node 121 determines whether the available storage capacity of the storage medium with the highest priority is greater than a threshold value based on the priorities of the multiple types of storage media, starting from the storage medium with the highest priority. If the available storage capacity of the storage medium with the highest priority is greater than the threshold value, it means that the storage medium with the highest priority has excess storage space to store processed data, and the storage medium with the highest priority is selected to store the processed data. If the available storage capacity of the storage medium with the highest priority is less than or equal to the threshold value, it means that the storage medium with the highest priority does not have excess storage space to store processed data, and then it is determined whether the available storage capacity of the storage medium with the second highest priority is greater than the threshold value, and the multiple types of storage media are traversed in turn, and finally the storage medium for storing the processed data is determined from the multiple types of storage media.

[0091] It should be noted that in actual applications, storage policies can be set according to business needs, scenario needs, or user needs. The above description of storage policies is just an example. For example, storage policies can also refer to data localization preferences, that is, preferentially storing processed data in local storage media in the global memory pool. For another example, storage policies can also refer to performance priority, storage capacity priority, cost priority, etc. according to the application's trade-off between performance and cost.

[0092] Before the computing node 121 uses the storage strategy to select a storage medium for storing the processed data, that is, before executing step 523 , the storage strategy and configuration information of various types of storage media may be preconfigured.

[0093] The storage strategy provided in the embodiment of the present application can be applied to at least one application supported by the above data processing system 100, namely, big data, database, high performance computing, artificial intelligence, distributed storage and cloud native. For example, the computing node 121 can use the storage strategy when selecting a storage medium for storing intermediate data in processing a big data business or a task in a big data business or a task in the whole system.

[0094] It should be noted that in the embodiment of the present application, after the computing node 121 performs memory operations on the global memory pool according to the processing request data, that is, writes the processed data into the global memory pool or reads the data from the global memory pool, it indicates that the processing request is completed. The global memory pool provides the ability to asynchronously write data to the persistence and capacity layers of the storage node, as well as pre-fetch and cache from the persistence and capacity layers. Subsequently, the computing node 121 or the storage node 131 can also read the processed data from the global memory pool. The data processing operations on the data stored in the global memory pool are as follows Figure 6 's explanation. Figure 6 A flowchart of another data processing method for a fusion system provided in an embodiment of the present application.

[0095] Step 610: The computing node 121 reads data from the global memory pool according to the first address and processes the data.

[0096] When the computing node needs data stored in the global memory pool to perform a task, the computing node can read the relevant data from the global memory pool. For example, computing node 121 is a computing node that performs a reduce task. Computing node 121 reads the processed data required to perform the reduce task from the global memory pool according to the first address. Computing node 121 can obtain the first address from the first address control node 122.

[0097] Step 620 , the computing node 121 reads data from the global memory pool according to the first address, and writes the data to the storage node 131 .

[0098] When the data stored in the global memory pool needs to be persisted, that is, when the data stored in the global memory pool needs to be moved to the storage node, the computing node 121 can read the relevant data from the global memory pool and write it to the storage node 131. The storage node 131 writes the processed data to the storage medium of the storage node 131 according to the memory operation instruction. Optionally, the computing node 121 can also save the data written to the storage node 131.

[0099] In other embodiments, the computing node 121 may also perform step 630, that is, pre-fetch data from the storage node 131 according to the memory operation instruction and store it in the global memory pool. Thus, the computing node 121 can obtain data as quickly as possible, shortening the end-to-end data processing time.

[0100] The computing node 121 may also execute step 640, that is, perform memory operations on data between the global memory pool and the storage node based on the memory operation instructions according to the hot and cold characteristics of the data.

[0101] Hot data refers to data that is accessed more frequently, such as online data. Cold data refers to data that is accessed less frequently, such as enterprise backup data, business and operation log data, call records and statistical data. Hot data has a high access frequency and high efficiency requirements, so it is calculated and deployed nearby. Cold data has a low access frequency and low efficiency requirements, so it can be deployed centrally.

[0102] The embodiment of the present application constructs a global memory pool under the storage separation architecture as a unified cache layer for computing and storage. The global memory pool integrates storage and computing, and realizes logical integration on the basis of physical separation. In this layer, computing nodes and storage nodes use unified memory semantics to exchange and share data, thereby avoiding intermediate operations such as repeated format conversion and protocol conversion between computing nodes and storage nodes, achieving extreme high performance and low energy consumption.

[0103] The following example illustrates the data processing process based on the fusion system. Figure 7 , which is a schematic diagram of a data processing process for executing map tasks and reduce tasks provided in an embodiment of the present application.

[0104] 1) Initialization phase: ① Load application data, that is, load the program configuration file and start the application. Application data can be stored in non-volatile memory or global memory pool, and the computing node loads the application data into the local memory of the computing node. For example, the computing node uses direct memory access (DMA) to load application data from non-volatile memory to local memory, or loads application data through a virtual file system (VFS) or portable operating system interface (POSIX) file system interface built on the memory level. 2○ Load the data to be processed. The data to be processed can be stored in non-volatile memory, and the computing node loads the data to be processed into the local memory of the computing node. For example, the computing node obtains the data to be processed from the non-volatile memory pool according to the memory operation instruction and loads it into the local memory, eliminating the access overhead of the file system stack. The computing node can also use remote direct memory access (RDMA) to obtain the data to be processed stored in the remote node, achieve efficient access, and reduce the CPU overhead and network protocol stack overhead of remote data access.

[0105] 2) Data processing stage: 3○ According to the processing request data, the memory operation instructions are used to process the data to be processed to obtain the processed data. For example, each computing node sorts the processed data to obtain the intermediate data. The computing node selects the storage medium for storing the processed data from the global memory pool according to the amount of processed data, for example, storing the processed data in HBM, DRAM or SCM. When the storage capacity in the global memory pool is insufficient, it can be solved by remote memory expansion or memory disk expansion, that is, storing the processed data in other storage media. The destination node reads the intermediate data from the source node or the source node writes the intermediate data to the destination node to complete the data exchange (shuffle). If the global memory pool is constructed by independent storage media (such as the storage medium of the computing node and the storage medium of the storage node), the source node can write the intermediate data to the global memory pool, and then the destination node reads the intermediate data from the global memory pool. 4○ Merge the sorting results.

[0106] 3) Persistence phase: ○5 The test report and merge sort results are written to the local non-volatile memory or the local memory of the global memory pool, and asynchronously written to the storage space of the storage node by the global memory pool component.

[0107] 4) Cleanup phase: ○6 Release local memory or global memory pool.

[0108] It is understandable that in order to implement the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0109] Combined with the above Figures 1 to 7 , describes in detail the data processing method of the fusion system provided by this embodiment, and will now be combined with Figure 8 , describing the data processing device provided according to this embodiment.

[0110] Figure 8 The following is a schematic diagram of the structure of a possible data processing device provided in this embodiment. These data processing devices can be used to implement the functions of the computing device or computing node in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the data processing device can be as follows: Figure 5 The computing node 121 shown may also be a module (such as a chip) applied to a server.

[0111] like Figure 8As shown, the data processing device 800 includes a communication module 810, a data processing module 820 and a storage module 830. The data processing device 800 is used to implement the above Figure 5 The function of computing node 121 in the method embodiment shown in FIG.

[0112] The communication module 810 is used to obtain processing request data.

[0113] The data processing module 820 is used to perform a memory operation on the global memory pool to process the requested data according to the memory operation instruction. For example, the data processing module 820 is used to execute Figure 5 Step 510 and step 520.

[0114] The storage module 830 is used to store memory operation instructions so that the data processing module 820 can obtain processed data. Figure 5 Medium S560.

[0115] The data processing module 820 is specifically used to read the data to be processed and the application data from the global memory pool; process the data to be processed using the memory operation instruction according to the processing request data to obtain processed data, and write the processed data into the storage space indicated by the first address in the global memory pool.

[0116] The data processing module 820 is further configured to read processed data from the global memory pool according to the first address.

[0117] The data processing module 820 is further configured to pre-fetch data from the storage node according to the memory operation instruction, and store the data in the global memory pool.

[0118] The data processing module 820 is further configured to perform memory operations on data between the global memory pool and the storage node based on the memory operation instructions according to the hot and cold characteristics of the data.

[0119] It should be understood that the data processing device 800 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. It can also be implemented by software. Figure 5 or Figure 6 The data processing method shown in the figure and its various modules may also be software modules, and the data processing device 800 and its various modules may also be software modules.

[0120] The data processing device 800 according to the embodiment of the present application may be corresponding to the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the data processing device 800 are respectively to implement Figure 5 or Figure 6 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0121] Fig. 9 A schematic diagram of the structure of a computing device 900 provided in this embodiment. As shown in the figure, the computing device 900 includes a processor 910, a bus 920, a memory 930, a communication interface 940, and a memory unit 950 (also referred to as a main memory unit). The processor 910, the memory 930, the memory unit 950, and the communication interface 940 are connected via a bus 920.

[0122] It should be understood that in this embodiment, the processor 910 may be a CPU, and the processor 910 may also be other general-purpose processors, digital signal processors (DSP), ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0123] The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0124] The communication interface 940 is used to implement the communication between the computing device 900 and an external device or component. Figure 5 When the computing node 121 functions as shown, the communication interface 940 is used to obtain processing request data, and the computing node 121 performs a memory operation on the processing request data on the global memory pool according to the memory operation instruction.

[0125] The bus 920 may include a path for transmitting information between the above components (such as the processor 910, the memory unit 950, and the memory 930). In addition to the data bus, the bus 920 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 920 in the figure. The bus 920 may be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a computer express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 920 can be divided into an address bus, a data bus, a control bus, etc.

[0126] As an example, the computing device 900 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. The processor here may refer to one or more devices, circuits, and / or computing units for processing data (such as computer program instructions). In this embodiment, the computing device 900 is used to implement Figure 5 When the computing node 121 functions as shown, the processor 910 may perform a memory operation on the processing request data on the global memory pool according to a memory operation instruction.

[0127] It is worth mentioning that Fig. 9 In the example, a computing device 900 including one processor 910 and one memory 930 is taken. Here, the processor 910 and the memory 930 are respectively used to indicate a type of device or equipment. In a specific embodiment, the number of each type of device or equipment can be determined according to business requirements.

[0128] The memory unit 950 may correspond to the global memory pool for storing information such as processed data in the above method embodiment. The memory unit 950 may be a volatile memory pool or a non-volatile memory pool, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0129] The memory 930 may correspond to a storage medium used to store information such as computer instructions, memory operation instructions, storage strategies, etc. in the above method embodiments, for example, a disk such as a mechanical hard disk or a solid-state hard disk.

[0130] The computing device 900 may be a general-purpose device or a dedicated device. For example, the computing device 900 may be an edge device (e.g., a box carrying a chip with processing capabilities), etc. Alternatively, the computing device 900 may also be a server or other device with computing capabilities.

[0131] It should be understood that the computing device 900 according to this embodiment may correspond to the data processing device 800 in this embodiment, and may correspond to the device executing the Figure 5 or Figure 6 The corresponding subject in any method, and the above and other operations and / or functions of each module in the data processing device 800 are respectively to implement Figure 5 or Figure 6 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0132] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist in a computing device as discrete components.

[0133] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing method for a fusion system, characterized in that: The fusion system includes a computing node and a storage node, the computing node is connected to the storage node through a network, and the storage medium of the computing node and the storage medium of the storage node are uniformly addressed to form a global memory pool; the method includes: Get processing request data; Obtain memory operation instructions according to processing request data; A memory operation is performed on the processing request data in the global memory pool according to the memory operation instruction.

2. The method according to claim 1, characterized in that: The memory operation instruction includes at least one of memory allocation, memory setting, memory copying, memory moving, memory releasing and memory comparison.

3. The method according to claim 2, characterized in that The storage medium of the global memory pool includes memory, memory server and storage-level memory SCM.

4. The method according to any one of claims 1 to 3, characterized in that Performing the memory operation of processing the requested data on the global memory pool according to the memory operation instruction includes: Read the data to be processed from the global memory pool; The data to be processed is processed using the memory operation instruction according to the processing request data to obtain processed data, and the processed data is written into the storage space indicated by the first address in the global memory pool.

5. The method according to claim 4, characterized in that Writing the processed data into a storage space indicated by a first address in the global memory pool includes: Determine, based on user requirements and storage medium characteristics, that the processed data is written into the storage space indicated by the first address in the global memory pool, wherein the storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio, and reliability.

6. The method according to claim 4 or 5, characterized in that: The storage space indicated by the first address includes one of a storage space provided by a storage medium of the computing node and a storage space provided by a storage medium of the storage node.

7. The method according to any one of claims 4 to 6, characterized in that: After writing the processed data into the storage space indicated by the first address in the global memory pool, the method further includes: The processed data is read from the global memory pool according to the first address.

8. The method according to claim 7, characterized in that Reading the processed data from the global memory pool according to the first address includes: The processed data is read from the global memory pool according to the first address, and the processed data is written into the storage node.

9. The method according to claim 1, characterized in that: The method further comprises: Pre-fetch data from the storage node according to the memory operation instruction, and store the data in the global memory pool.

10. The method according to claim 1, characterized in that The method further comprises: According to the hot and cold characteristics of the data, memory operations on the data are performed between the global memory pool and the storage node based on the memory operation instructions.

11. A data processing device for a fusion system, characterized in that: The fusion system includes a computing node and a storage node, the computing node is connected to the storage node through a network, and the storage medium of the computing node and the storage medium of the storage node are uniformly addressed to form a global memory pool; the device includes: A communication module, used to obtain processing request data; The data processing module is used to obtain a memory operation instruction according to the processing request data, and perform a memory operation on the processing request data in the global memory pool according to the memory operation instruction.

12. The device according to claim 11, characterized in that The memory operation instruction includes at least one of memory allocation, memory setting, memory copying, memory moving, memory releasing and memory comparison.

13. The device according to claim 12, characterized in that The storage medium of the global memory pool includes memory, memory server and storage-level memory SCM.

14. The device according to any one of claims 11 to 13, characterized in that When the data processing module performs the memory operation of the processing request data on the global memory pool according to the memory operation instruction, it is specifically used to: Read the data to be processed from the global memory pool; The data to be processed is processed using the memory operation instruction according to the processing request data to obtain processed data, and the processed data is written into the storage space indicated by the first address in the global memory pool.

15. The device according to claim 14, characterized in that When the data processing module writes the processed data into the storage space indicated by the first address in the global memory pool, it is specifically used to: Determine, based on user requirements and storage medium characteristics, that the processed data is written into the storage space indicated by the first address in the global memory pool, wherein the storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio, and reliability.

16. The device according to claim 14 or 15, characterized in that The storage space indicated by the first address includes one of a storage space provided by a storage medium of the computing node and a storage space provided by a storage medium of the storage node.

17. The device according to any one of claims 14 to 16, characterized in that The data processing module is further used to read the processed data from the global memory pool according to the first address.

18. The device according to claim 17, characterized in that When the data processing module reads the processed data from the global memory pool according to the first address, it is specifically used to: The processed data is read from the global memory pool according to the first address, and the processed data is written into the storage node.

19. The device according to claim 11, characterized in that The data processing module is further used to pre-fetch data from the storage node according to the memory operation instruction and store the data in the global memory pool.

20. The device according to claim 11, characterized in that The data processing module is further used to perform memory operations on data between the global memory pool and the storage node based on the memory operation instructions according to the hot and cold characteristics of the data.

21. A computing device, characterized in that The computing device includes a memory and at least one processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method described in any one of claims 1 to 10 are performed.

22. A system, characterized in that: The system includes a storage node and a computing node, the computing node is connected to the storage node via a network, the storage medium of the computing node and the storage medium of the storage node are uniformly addressed to form a global memory pool, and the computing node is used to execute the operating steps of the method described in any one of claims 1 to 10.

Images