Data processing method, device, equipment and system of fusion system

Abstract

A data processing method, device and system of a fusion system are disclosed, and relate to the field of data. The fusion system comprises a computing node and a storage node. The computing node is connected with the storage node through a network to construct a storage-computing separation architecture. The storage medium of the computing node and the storage medium of the storage node are uniformly addressed to constitute a global memory pool, that is, a global storage medium shared by the computing node and the storage node. When performing read and write operations on the system, processing request data is obtained, and a memory operation of the processing request data is performed on the global memory pool according to a memory operation instruction. Thus, based on the storage-computing separation architecture with the global memory pool, the computing and storage can be flexibly expanded as needed, the read and write operations on the system can be quickly performed, the end-to-end data processing time is shortened, the data transmission speed is improved, the system energy consumption is reduced, and the system performance bottleneck is improved.

Description

[0001] This application claims priority to the Chinese Patent Application No. 202210369243.7, filed on April 8, 2022, entitled "A Fusion System of Computing and Storage", the content of which is incorporated herein by reference in its entirety.

[0002] This application is a divisional application of the original application with the application number 202210633694.7 and the original filing date of June 6, 2022, the content of the original application is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of data, in particular to a data processing method, device, equipment and system of a fusion system. BACKGROUND

[0004] Storage and computing are two key systems of distributed applications (such as big data, database), which determine the overall performance and energy consumption of the system. Generally, computing and storage are deployed on the same node, and the system performance is high, but the storage-to-computing ratio is determined, and computing and storage cannot be flexibly expanded separately. Subsequently, the storage-computing separation architecture emerged, and the storage cluster and the computing cluster are connected through the network to achieve the flexible expansion of computing and storage according to demand. However, data needs to go through multiple conversion operations such as protocols and formats from the computing side to the storage side, resulting in a long data processing time and high system energy consumption, and system performance has become a bottleneck. SUMMARY

[0005] The present application provides a data processing method, device, equipment and system of a fusion system, thereby shortening the data processing time, improving the data transmission speed, and reducing the system energy consumption.

[0006] In a first aspect, a data processing method of a fusion system is provided. The fusion system includes a computing node and a storage node. The computing node and the storage node are connected through a network to build a storage-computing separation architecture. The storage medium of the computing node and the storage medium of the storage node form a global memory pool through unified addressing, i.e., a global memory shared by the computing node and the storage node. When performing read and write operations on the system, the processing request data is obtained, and the global memory pool is processed according to the memory operation instruction of the processing request data.

[0007] Thus, since the memory operation instruction is a technology of performing memory operations on the memory by using the memory interface, the memory operation on the global memory shared by the computing node and the storage node according to the memory operation instruction avoids the data from the computing side to the storage side needing to undergo multiple conversion operations such as protocols and formats, effectively simplifying the data moving process between the computing node and the storage node. Thus, based on the storage-computing separation architecture with the global memory pool, the computing and storage can be flexibly expanded on demand, and the read-write operation of the system can be quickly performed, the end-to-end data processing time is shortened, the data transmission speed is improved, the system energy consumption is reduced, and the system performance bottleneck is improved.

[0008] The memory operation instruction includes at least one of memory allocation, memory setting, memory copying, memory moving, memory releasing, and memory comparison. The embodiments of the present application do not limit the type of the storage medium of the global memory pool. The storage medium of the global memory pool includes memory, hard disk, memory server, and storage-class memory (SCM). Thus, the introduction of the global memory pool including multiple storage media in the distributed processing system makes the storage medium of the stored data have more possibilities, selects the matching storage medium to store the data, and realizes the read-write operation of the system, improves the data transmission speed, and shortens the end-to-end data processing time.

[0009] In a possible implementation, the memory operation on the global memory pool according to the memory operation instruction includes: reading the to-be-processed data from the global memory pool; processing the to-be-processed data according to the processing request data using the memory operation instruction to obtain processed data, and writing the processed data into the storage space indicated by the first address in the global memory pool. The storage space indicated by the first address includes one of the storage space provided by the storage medium of the computing node and the storage space provided by the storage medium of the storage node.

[0010] The computing node can determine the storage space indicated by the first address according to the storage strategy. For example, the computing node determines the storage space indicated by the first address according to the access characteristics of the application.

[0011] Example one, according to the user demand and the storage medium characteristics to determine the writing of the processed data into the storage space indicated by the first address in the global memory pool. The storage medium characteristics include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, central processing unit (CPU) consumption, energy consumption ratio, and reliability. Thus, based on the user demand and the storage medium characteristics of the system to perform the read-write operation of the system, not only enhances the user's control authority of the system, but also improves the user's system experience and expands the application scenarios of the system.

[0012] In the second example, determining the storage space in the first address of the global memory pool to which the processed data is to be written according to the user demand and the storage medium characteristics comprises: determining the storage space in the first address of the computing node of the global memory pool to which the processed data is to be written according to the user demand and the storage medium characteristics. Thus, the processed data is stored in the local memory of the computing side of the global memory pool, so as to perform local memory access, effectively shorten the data processing time, and improve the data transmission speed.

[0013] In the third example, determining the storage space in the first address of the global memory pool to which the processed data is to be written according to the user demand and the storage medium characteristics comprises: determining the storage space in the first address of the storage node of the global memory pool to which the processed data is to be written according to the user demand and the storage medium characteristics. Thus, the processed data is stored in the memory of the storage side of the global memory pool, and the reliability and persistence of the data are improved.

[0014] In another possible implementation, after the processed data is written into the storage space in the first address of the global memory pool, the method further comprises: reading the processed data from the global memory pool according to the first address.

[0015] For example, when the data stored in the global memory pool is to be written to a disk, reading the processed data from the global memory pool according to the first address comprises: when the processed data needs to be persisted, the computing node reads the processed data from the global memory pool according to the first address, and writes the processed data into the storage node.

[0016] For another example, when the computing node in the fusion system uses the processed data, the computing node reads the processed data from the global memory pool according to the first address.

[0017] In another possible implementation, the method further comprises: when the processed data is read from the global memory pool according to the first address, performing another memory operation on the global memory pool according to the memory operation instruction.

[0018] Thus, from the perspective of the application, the efficiency of data processing and the data transmission speed are improved, and from the perspective of the hardware device, the utilization rate of system resources is improved.

[0019] In another possible implementation, the method further comprises: pre-fetching data from the storage node according to the memory operation instruction, and storing the data into the global memory pool. Thus, the computing node can quickly obtain the data, and the data processing time is shortened.

[0020] In another possible implementation, the method further includes: performing, according to data hot and cold characteristics, memory operations of data between the global memory pool and the storage node based on the memory operation instruction. Cold data refers to data with a low access frequency. Moving the cold data from the global memory pool to the storage node releases storage space of the global memory pool, and improves utilization of the storage space of the global memory pool. Hot data refers to data with a high access frequency. Moving the hot data from the storage node to the global memory pool enables the computing node to obtain required data as soon as possible, shortens a data processing time length, and reduces computing resources occupied by frequent reading and writing of data.

[0021] In a second aspect, a data processing apparatus of a fusion system is provided, and the apparatus includes various modules for performing the data processing method of the fusion system in the first aspect or any possible design in the first aspect.

[0022] In a third aspect, a computing device is provided, which includes at least one processor and a memory, and the memory is configured to store a set of computer instructions; when the processor executes the set of computer instructions as the computing node in the first aspect or any possible implementation of the first aspect, the operation steps of the data processing method of the fusion system in the first aspect or any possible implementation of the first aspect are performed.

[0023] In a fourth aspect, a computer-readable storage medium is provided, which includes: computer software instructions; when the computer software instructions are run in a computing device, the computing device is caused to perform the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0024] In a fifth aspect, a computer program product is provided, which, when run on a computer, causes a computing device to perform the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0025] On the basis of the implementation manners of the aspects provided in the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 An architecture schematic diagram of a data processing system provided by an embodiment of the present application;

[0027] Figure 2 A schematic diagram of data distributed processing provided by an embodiment of the present application;

[0028] Figure 3 A deployment scene schematic diagram of a global memory pool provided by an embodiment of the present application;

[0029] Figure 4A three-layer structure storage system schematic diagram provided for an embodiment of the present application;

[0030] Figure 5 A data processing method process schematic diagram provided for an embodiment of the present application;

[0031] Figure 6 A data processing method process schematic diagram provided for another embodiment of the present application;

[0032] Figure 7 A Map task and Reduce task processing process schematic diagram based on a global memory pool provided for an embodiment of the present application;

[0033] Figure 8 A data processing apparatus structure schematic diagram provided for an embodiment of the present application;

[0034] Figure 9 A computing device structure schematic diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0035] With the development of Internet, Internet of Things, network bandwidth, intelligent terminal and cloud computing services, the data type and data scale develop at an unprecedented speed, and big data, database, high performance computing (HPC) and other applications are generated, and data changes from a single processing object to a basic resource.

[0036] Big data is a collection of data that cannot be captured, managed and processed in a time range by conventional software tools. Since the large amount of data contained in big data has a correlation relationship, the data relationship in big data is mined by using a data analysis method, model or tool to analyze big data, and the data relationship of big data is used for prediction or decision making. For example, analyzing user shopping trend data, pushing items that the user may purchase to the user, and improving the user's shopping experience. Therefore, big data has the characteristics of large data volume, fast data growth rate, diverse data types and high utilization value.

[0037] A database is a computer software system that stores and manages data according to a data structure. It can also be understood as a collection of data in a computer for storing and managing a large amount of data, that is, an electronic file cabinet. Data can include travel records, consumption records, browsed web pages, sent messages, images, music and sounds, etc.

[0038] Since the data volume of these applications is very large, a single computing node cannot meet the computing demand. Usually, a distributed data processing system is used to process data.

[0039] Figure 1An architecture diagram of a data processing system is provided in embodiments of the present application. As shown in Figure 1 The data processing system 100 includes a client 110, a computing cluster 120, and a storage cluster 130.

[0040] The storage cluster 130 includes at least two storage nodes 131. One storage node 131 includes one or more controllers, network cards, and a plurality of hard disks. The hard disks are used to store data. The hard disks can be magnetic disks or other types of storage media, such as solid state disks or shingled magnetic recording hard disks, etc. The network cards are used to communicate with the computing nodes 121 included in the computing cluster 120. The controllers are used to write data into the hard disks or read data from the hard disks according to read / write data requests sent by the computing nodes 121. In the process of reading and writing data, the controllers need to convert the addresses carried in the read / write data requests into addresses that can be recognized by the hard disks.

[0041] The client 110 communicates with the computing cluster 120 and the storage cluster 130 through a network 140. For example, the client 110 sends a service request to the computing cluster 120 through the network 140, requesting the computing cluster 120 to perform distributed processing on service data included in the service request. The network 140 can be an enterprise intranet (such as a local area network (Local Area Network, LAN)) or the Internet.

[0042] The data processing system 100 supports running big data, database, high-performance computing, artificial intelligence, distributed storage, and cloud-native applications. Understandably, the service data in embodiments of the present application includes data of big data, database, high-performance computing, artificial intelligence (Artificial Intelligence, AI), distributed storage, and cloud-native applications.

[0043] In some embodiments, the storage cluster 130 stores and manages a large amount of data based on a distributed file system 132 and a distributed database 133. The client 110 is installed with a client program 111, and the client 110 runs the client program 111 to display a user interface (user interface, UI). The user 150 operates the user interface to access the distributed file system 132 and the distributed database 133 to obtain data and instruct the computing cluster 120 to process data services. The client 110 can be a computer connected to the network 140, which can also be referred to as a workstation. Different clients can share resources (such as computing resources and storage resources) on the network.

[0044] The computing cluster 120 includes at least two computing nodes 121, which can communicate with each other. The computing node 121 is a computing device, such as a server, a desktop computer, or a controller of a storage array, etc. For example, the big data service submitted by the client 110 can be referred to as a job. The job can be divided into multiple tasks, which are executed in parallel by multiple computing nodes, and all the tasks end when the job is completed. The task is generally a part of data or a processing process in a stage in a job, and all the tasks are scheduled to be completed in parallel or in series. In some embodiments, the computing cluster 120 performs distributed processing on the big data service based on a MapReduce model 134. The MapReduce model is a distributed programming model, that is, the big data service is divided into map tasks and reduce tasks, the map tasks are executed by multiple computing nodes 121, and the processing results are collected to execute the reduce tasks. In some embodiments, as shown in FIG. 1, the computing cluster 120 includes a control node 122 and at least two computing nodes 121. The control node and the computing node can be independent physical devices, and the control node can also be referred to as a control device or a naming node. The computing node can be referred to as a computing device or a data node. The control node 122 is used to manage the namespace of the distributed file system and the access of the client 110 to the distributed file system. In addition, the control node 122 instructs the computing nodes to execute the map tasks and the reduce tasks. The computing nodes execute the map tasks based on the MapReduce model 134 to obtain intermediate data (also referred to as map data or shuffle data), and store the intermediate data in the global memory pool 170. The computing nodes that execute the reduce tasks read the intermediate data from the global memory pool 170. Alternatively, the storage cluster 130 can also process the data according to the MapReduce model 134 when storing the data. Figure 2

[0045] In other embodiments, the system administrator 160 can access the distributed file system 132 and the distributed database 133 through the client 110 to call the application platform interface (API) 112 or the command-line interface (CLI) interface 113 or the graphical user interface (GUI) to configure system information, etc. For example, the deployment information and the storage strategy of the global memory pool 170 configured for the computing nodes 121 and the storage nodes 131 through the uniform addressing provided by the embodiments of the present application.

[0046] ​The embodiment of the present application provides a memory operation instruction-based data processing method of a memory-computing integration architecture, that is, a global memory pool formed by uniformly addressing storage media of a computing node and storage media of a storage node, and the global memory pool is processed according to a memory operation instruction to perform a memory operation of requesting data. Thus, the flexibility of separately expanding the memory-computing integration is solved, and the performance of the memory-computing separation is also solved. The memory operation of the global memory pool according to the memory operation instruction includes a read operation and a write operation. The data processing system configured with the global memory pool can also be referred to as a fusion system. The memory operation instruction is a technology of performing a memory operation on the storage media by using a memory interface. Since the computing side and the storage side are two different memory operations, the memory operation instruction is used to perform the fusion of computing and storage based on the global memory pool uniformly arranged globally, that is, the memory operation of requesting data is performed on the global storage media shared by the computing node and the storage node according to the memory operation instruction. The data moving process between the computing node and the storage node is effectively simplified, and the data needs to be converted in multiple protocols and formats when moving from the computing side to the storage side, thereby ensuring that the computing and storage are flexibly expanded on demand based on the memory-computing separation architecture with the global memory pool, quickly performing the read and write operations of the system, shortening the end-to-end data processing time, improving the data transmission speed, reducing the system energy consumption, and improving the system performance bottleneck.

[0047] For example, the format conversion includes serialization and deserialization. Serialization refers to the process of converting an object into a byte sequence. Deserialization refers to the process of restoring a byte sequence into an object.

[0048] It should be noted that the global memory pool provided by the embodiment of the present application can include storage media of a computing node and storage media of a storage node in a data processing system. The storage media of the computing node includes at least one of a local storage medium in the computing node and an extended storage medium connected to the computing node. The storage media of the storage node includes at least one of a local storage medium in the storage node and an extended storage medium connected to the storage node.

[0049] For example, the global memory pool includes the local storage medium in the computing node and the local storage medium in the storage node.

[0050] For another example, the global memory pool includes the local storage medium in the computing node, the extended storage medium connected to the computing node, and any one of the local storage medium in the storage node and the extended storage medium connected to the storage node.

[0051] For another example, the global memory pool includes the local storage medium in the computing node, the extended storage medium connected to the computing node, the local storage medium in the storage node, and the extended storage medium connected to the storage node.

[0052] For example, as shown in Figure 3 The deployment scenario of the global memory pool provided by the embodiment of the present application is shown. The global memory pool 300 includes the storage medium 310 in each of the N computing nodes, the extended storage medium 320 connected to each of the N computing nodes, the storage medium 330 in each of the M storage nodes, and the extended storage medium 340 connected to each of the M storage nodes.

[0053] It should be understood that the storage capacity of the global memory pool can include part of the storage capacity in the storage medium of the computing node and part of the storage capacity in the storage medium of the storage node. The global memory pool is the storage medium that can be accessed by the computing node and the storage node through unified addressing. The storage capacity of the global memory pool can be used by the computing node or the storage node through the memory interface of the large memory, the distributed data structure, the data cache, the metadata, etc. The computing node running the application program can use these memory interfaces to perform memory operations on the global memory pool. In this way, the global memory pool constructed based on the storage capacity of the storage medium of the computing node and the storage medium of the storage node provides a unified memory interface for the computing node in the north direction, so that the computing node uses the unified memory interface to write data into the storage space provided by the computing node of the global memory pool or the storage space provided by the storage node, and realizes the calculation and storage of the data based on the memory operation instruction.

[0054] The above is an example of constructing the global memory pool by the storage medium in the computing node and the storage medium in the storage node. The deployment mode of the global memory pool can be flexible and varied, and the embodiment of the present application is not limited. For example, the global memory pool is constructed by the storage medium of the storage node. For example, the global cache of high performance. For another example, the global memory pool is constructed by the storage medium of the computing node. For example, the global cache of high performance. Using the storage medium of the separate storage node or the storage medium of the computing node to construct the global memory pool can reduce the occupation of the storage resource on the storage side, and provide a more flexible expansion scheme.

[0055] It should be noted that the memory is a memory device for storing programs and various data. The larger the capacity of the memory, the slower the access speed. Conversely, the smaller the capacity of the memory, the faster the access speed. The access speed refers to the data transmission speed when data is written or read to the memory. The access speed can also be referred to as the read-write speed. According to the storage capacity and the access speed, the memory can be divided into different levels.

[0056] For example, Figure 4 The storage system with a three-layer structure provided by the embodiment of the present application is shown. From the first layer to the third layer, the storage capacity increases gradually, the access speed decreases gradually, and the cost decreases gradually. For example, Figure 4As shown, the first level includes registers 411, a level one cache 412, a level two cache 413 and a level three cache 414 in a central processing unit (CPU). The second level includes memories that can serve as the main memory of the computer system. For example, a dynamic random access memory (DRAM) 421, a double data rate synchronous dynamic random access memory (DDR SDRAM) 422, a storage-class-memory (SCM) 423. The main memory can be referred to as the main memory or the memory, i.e., the memory that exchanges information with the CPU. The third level includes memories that can serve as the auxiliary memory of the computer system. For example, a network memory 431, a solid state disk (SSD) 432, a hard disk drive (HDD) 433. The auxiliary memory can be referred to as the auxiliary storage or the external storage. Compared with the main memory, the external storage has a larger storage capacity and a slower access speed. It can be seen that the closer the memory is to the CPU, the smaller the capacity, the faster the access speed, the larger the bandwidth and the lower the delay. Therefore, the memory included in the third level stores data that is not frequently accessed by the CPU, thereby improving the reliability of the data. The memory included in the second level can serve as a cache device for storing data frequently accessed by the CPU, thereby significantly improving the access performance of the system.

[0057] The storage medium of the global memory pool provided in the embodiments of the present application includes a memory (such as a DRAM), an SSD, a hard disk, a memory server and an SCM.

[0058] In some embodiments, the global memory pool can be set according to the type of the storage medium, i.e., a type of storage medium is used to construct a type of memory pool, different types of storage medium are used to construct different types of global memory pool, the global memory pool is applied to different scenarios, the computing node selects the storage medium according to the access characteristics of the application, the user's control authority over the system is enhanced, the user's system experience is improved, and the application scenarios applicable to the system are expanded. For example, the DRAM in the computing node and the DRAM in the storage node are uniformly addressed to form a DRAM memory pool. The DRAM memory pool is used for application scenarios that have high requirements on access performance, moderate data capacity and no data persistence appeal. For another example, the SCM in the computing node and the SCM in the storage node are uniformly addressed to form a SCM memory pool. The SCM memory pool is used for application scenarios that are not sensitive to access performance, have large data capacity and have appeal for data persistence.

[0059] Different storage media have different storage media characteristics. The storage media characteristics include at least one of a write latency, a read latency, a total storage capacity, an available storage capacity, an access speed, a CPU consumption, an energy consumption ratio, and a reliability. The write latency refers to a latency of the computing node 121 writing data to the storage media. The read latency refers to a latency of the computing node 121 reading data from the storage media. The storage capacity refers to a total storage capacity of the storage media that can store data. The available storage capacity refers to a remaining storage capacity of the total storage capacity minus a used storage capacity. The access speed refers to a speed of the computing node 121 performing read and write operations on the storage media. The CPU consumption refers to an occupancy rate of a CPU of the computing node 121 used for writing data to the storage media or reading data from the storage media. The energy consumption ratio refers to energy (such as electric energy) consumed per unit of time. The reliability refers to a persistence degree of the storage media storing data.

[0060] The embodiments of the data processing method of the fusion system provided by the present application will be described in detail below with reference to the accompanying drawings.

[0061] Figure 5 A flowchart of the data processing method of the fusion system provided by the present application is shown. Here, the client 110, the computing cluster 120, and the storage cluster 130 are taken as examples for description. The computing cluster 120 includes the control node 122 and at least two computing nodes 121, and the storage cluster 130 includes at least two storage nodes 131. The control node 122 is configured to control the computing nodes 121 to perform distributed processing of data. The computing nodes 121 and the storage nodes 131 are configured with a global memory pool 170, and the global memory pool 170 includes storage media of the computing nodes 121 and storage media of the storage nodes 131. As shown in the figure, the method includes the following steps. Figure 5

[0062] In step 510, the computing node 121 acquires processing request data.

[0063] The client 110 responds to a user operation and sends a service request of a data service to the control node 122. The control node 122 can receive the service request of the large data service sent by the client 110 through a local area network or the Internet. The service request can include a service identifier and service data. The service identifier is used to uniquely indicate a data service. The service data can be data for which the computing node performs distributed processing of data or identifier data indicating data to be processed.

[0064] ​The user operation can refer to an operation of operating a big data user interface to submit a big data service. The big data service includes a data analysis service, a data query service, a data modification service, and the like. For example, the big data service refers to analyzing personal data and purchase behavior data of a customer to depict a user portrait to classify the customer, so that a specific product or preferential product can be recommended to the specific customer, customer satisfaction is improved, and customer relationship is stabilized, and the like. For another example, the big data service refers to analyzing historical sales volume of a product to predict future sales volume, finding a reason for a decrease in sales volume or a reason for an increase in sales volume, and recommending a constructive suggestion to improve the sales volume.

[0065] The user operation can also refer to an operation of operating a database user interface to submit a database service, and the like. The database operation includes database creation, deletion, modification, and query, and the like.

[0066] The computing node 121 can receive the service request sent by the control node 122, convert the service request into processing request data conforming to an operation rule of the memory operation instruction, so as to perform, by the computing node 121, a memory operation of the processing request data on the global memory pool according to the memory operation instruction.

[0067] In step 520, the computing node 121 performs a memory operation of the processing request data on the global memory pool according to the memory operation instruction.

[0068] The memory operation instruction can also be referred to as a memory semantic or a memory operation function. The memory operation instruction includes at least one of a memory allocation (malloc), a memory setting (memset), a memory copy (memcpy), a memory move (memmove), a memory release (memory release), and a memory comparison (memcmp).

[0069] The memory allocation is used to support an application to run and allocate a piece of memory.

[0070] The memory setting is used to set a data mode of the global memory pool, for example, initialization.

[0071] The memory copy is used to copy data stored in a storage space indicated by a source address (source) to a storage space indicated by a destination address (destination).

[0072] The memory move is used to copy data stored in a storage space indicated by a source address (source) to a storage space indicated by a destination address (destination), and delete the data stored in the storage space indicated by the source address (source).

[0073] The memory comparison is used to compare whether data stored in two storage spaces are equal.

[0074] The memory releasing is used for releasing data stored in the memory, so as to improve utilization of system memory resources and improve system performance.

[0075] The processing request data is used for indicating an operation on the to-be-processed data. For example, the processing request data indicates obtaining a first quarter sales volume of product A. The to-be-processed data can include an annual sales volume. The specific operation process included in step 520 is described as follows.

[0076] In step 521, the computing node 121 reads the to-be-processed data and the application data.

[0077] The to-be-processed data and the application data can be stored in a storage space such as a global memory pool, a storage medium of a storage node, or a storage medium of a computing node. The computing node 121 can read the to-be-processed data and the application data from the global memory pool, the storage medium of the storage node, or the storage medium of the computing node. For example, the computing node 121 reads the application data from a local memory and reads the to-be-processed data from the global memory pool. The to-be-processed data can be an object that needs to be processed according to the indication of the processing request data. The application data includes an application program and application configuration data.

[0078] In step 522, the computing node 121 starts the application program according to the application data, and processes the to-be-processed data according to the processing request data to obtain processed data.

[0079] For example, it is assumed that the application data includes a database application program and database configuration data, and the processing request data is used for indicating sorting of Terabyte (TB) to-be-processed data. The computing node 121 starts the database according to the database application data, obtains the to-be-processed data, and sorts the to-be-processed data.

[0080] It should be noted that the computing node 121 can start the application program according to the application data before obtaining the processing request data, or can start the application program according to the application data after obtaining the processing request data.

[0081] In this way, the data processing based on the memory operation instruction can be applied to the whole life cycle of data processing. The data processing using the memory operation instruction realizes de-Input / Output (IO) and extremely fast access of data end to end, avoids format and protocol conversion in execution of data migration, and effectively reduces the data processing time length end to end.

[0082] In step 523, the computing node 121 writes the processed data into a storage space indicated by a first address in the global memory pool.

[0083] The computing node 121 automatically selects a storage medium from the global memory pool for storing the processed data according to a storage strategy. The storage strategy includes an applied access characteristic and a storage medium characteristic in the global memory pool. The storage space indicated by the first address includes one of a storage space provided by a storage medium of the computing node 121 and a storage space provided by a storage medium of the storage node 131.

[0084] In some embodiments, the computing node 121 determines to write the processed data into the storage space indicated by the first address in the global memory pool according to a user demand and a storage medium characteristic. The user demand is used to indicate a demand related to the storage medium characteristic. The processing request data includes the user demand. The storage medium characteristic includes at least one of a write latency, a read latency, a total storage capacity, an available storage capacity, an access speed, a central processing unit (CPU) consumption, an energy consumption ratio, and a reliability.

[0085] For example, the computing node 121 is configured with storage medium characteristics of multiple types of storage media. The user demand indicates an access speed range or a specific access speed, and the computing node 121 determines a storage medium from the global memory pool that meets the user demand. For example, the user demand indicates an access speed of a memory. The computing node 121 selects a storage medium from the global memory pool that meets the access speed of the memory, such as at least one of a memory, a DRAM, and an SCM. In example one, the computing node 121 determines to write the processed data into the storage space of the computing node indicated by the first address in the global memory pool according to the user demand and the storage medium characteristic. The storage space of the computing node indicated by the first address meets the access speed indicated by the user demand. Thus, the processed data is stored in the local memory of the computing side in the global memory pool, so as to perform local memory access, effectively shorten the data processing time, and improve the data transmission speed.

[0086] For another example, the computing node 121 is configured with an association relationship between a storage medium and a customer level. The user demand indicates a first customer level. The computing node 121 determines a storage medium associated with the first customer level from the association relationship according to the first customer level, and determines that the storage medium associated with the first customer level is used to store the processed data. In example two, the computing node 121 determines to write the processed data into the storage space of the storage node indicated by the first address in the global memory pool according to the user demand and the storage medium characteristic. The storage space of the computing node indicated by the first address meets the first customer level indicated by the user demand.

[0087] In this way, when the computing node 121 selects a storage medium from the global memory pool for storing the processed data, the computing node 121 dynamically selects a storage medium that meets the user demand based on the user demand of the user for the storage medium characteristic such as the access speed or the reliability, so as to ensure the performance and the scenario demand of the reliability of data processing.

[0088] In some embodiments, the computing node 121 is configured to determine the storage medium for storing the processed data according to the priority of the plurality of types of storage media determined according to the characteristics of the storage media, and the priority of the plurality of types of storage media indicated by the storage strategy. The priority of the plurality of types of storage media can be determined according to the access speed of the storage media. For example, the access speed of the memory is higher than that of the hard disk, and the access speed of the hard disk is higher than that of the extended storage medium. The priority of the plurality of types of storage media can be determined according to the priority of the deployment mode. For example, the priority of the local storage medium is higher than that of the extended storage medium. The priority of the plurality of types of storage media can be determined according to the characteristics of the plurality of types of storage media. The priority of the plurality of types of storage media is determined according to the priority of the deployment mode (e.g., local storage medium and extended storage medium). The priority of the storage medium under the same deployment mode can be determined according to the access speed of the storage medium.

[0089] Optionally, the computing node 121 can also select a storage medium with available storage capacity greater than a threshold from the plurality of types of storage media satisfying the user demand as the storage medium for storing the processed data.

[0090] The computing node 121 determines whether the available storage capacity of the storage medium with the highest priority is greater than a threshold according to the priority of the plurality of types of storage media, starting from the storage medium with the highest priority. If the available storage capacity of the storage medium with the highest priority is greater than the threshold, it indicates that the storage medium with the highest priority has excess storage space to store the processed data, and the storage medium with the highest priority is selected to store the processed data. If the available storage capacity of the storage medium with the highest priority is less than or equal to the threshold, it indicates that the storage medium with the highest priority does not have excess storage space to store the processed data, and the available storage capacity of the storage medium with the second highest priority is determined, and the plurality of types of storage media are traversed in turn, and finally the storage medium for storing the processed data is determined from the plurality of types of storage media.

[0091] It should be noted that in actual application, the storage strategy can be set according to business demand, scene demand or user demand, and the above description of the storage strategy is only illustrative. For example, the storage strategy can also refer to data localization preference, that is, the processed data is preferentially stored in the local storage medium in the global memory pool. For another example, the storage strategy can also refer to the selection of performance priority, storage capacity priority, cost priority, etc. according to the trade-off between performance and cost of the application.

[0092] Before the computing node 121 selects the storage medium for storing the processed data using the storage strategy, that is, before step 523 is performed, the storage strategy and the configuration information of the plurality of types of storage media can be pre-configured.

[0093] The storage strategy provided by the embodiments of the present application can be applied to at least one application supported by the data processing system 100, that is, big data, database, high-performance computing, artificial intelligence, distributed storage, and cloud native. For example, the computing node 121 can use the storage strategy when selecting a storage medium for storing intermediate data in processing big data services or processing tasks of big data services or processing system-wide tasks.

[0094] It should be noted that, after the computing node 121 performs memory operations on the global memory pool according to the processing request data, that is, after the processed data is written to the global memory pool or the data is read from the global memory pool, it indicates that the current processing request is completed. The global memory pool provides the ability to asynchronously write data to the persistence and capacity layer of the storage node, and to prefetch and cache from the persistence and capacity layer. Subsequently, the computing node 121 or the storage node 131 can also read the processed data from the global memory pool. The data processing operation on the data stored in the global memory pool is described as follows. Figure 6 Figure 6 The flowchart of another data processing method of a fusion system provided by the embodiments of the present application.

[0095] In step 610, the computing node 121 reads data from the global memory pool according to the first address and processes the data.

[0096] When the computing node needs to execute a task requiring data stored in the global memory pool, the computing node can read the related data from the global memory pool. For example, the computing node 121 is a computing node for executing a reduce task. The computing node 121 reads the processed data required for executing the reduce task from the global memory pool according to the first address. The computing node 121 can obtain the first address from the first address control node 122.

[0097] In step 620, the computing node 121 reads data from the global memory pool according to the first address and writes the data to the storage node 131.

[0098] When the data stored in the global memory pool needs to be persisted, that is, the data stored in the global memory pool needs to be moved to the storage node, the computing node 121 can read the related data from the global memory pool and write it to the storage node 131. The storage node 131 writes the processed data to the storage medium of the storage node 131 according to the memory operation instruction. Optionally, the computing node 121 can also save the data written to the storage node 131.

[0099] In other embodiments, the computing node 121 can also perform step 630, that is, according to the memory operation instruction, the data is prefetched from the storage node 131 and stored in the global memory pool. Thus, so as to facilitate the computing node 121 to obtain the data as soon as possible, and shorten the end-to-end data processing time. ​

[0100] The computing node 121 can further perform step 640, that is, performing memory operation of data between the global memory pool and the storage node based on the memory operation instruction according to the data hot and cold characteristics.

[0101] Hot data refers to data with high access frequency, such as online data. Cold data refers to data with low access frequency, such as enterprise backup data, business and operation log data, and bill and statistical data. Hot data has high efficiency requirement due to high access frequency requirement, and therefore is deployed close to the computing node, while cold data has low efficiency requirement due to low access frequency requirement, and therefore can be deployed centrally.

[0102] The embodiment of the present application constructs a global memory pool under a storage separation architecture, which is a cache layer for computing and storage unification. The global memory pool realizes logical fusion on the basis of physical separation of storage and computing, and the computing node and the storage node exchange and share data by using unified memory semantics, so that repeated intermediate operations such as format conversion and protocol conversion between the computing node and the storage node are avoided, and the highest performance and the lowest energy consumption are achieved.

[0103] The following illustrates the process of data processing based on the fusion system. As shown in FIG. 1, it is a schematic diagram of a data processing process for performing a map task and a reduce task according to an embodiment of the present application. Figure 7

[0104] 1) Initialization phase: ① Load application data, that is, load the program configuration file and start the application program. The application data can be stored in the non-volatile memory or the global memory pool, and the computing node loads the application data into the local memory of the computing node. For example, the computing node loads the application data from the non-volatile memory to the local memory by using the direct memory access (DMA) mode, or loads the application data by using the virtual file system (VFS) or the portable operating system interface (POSIX) file system interface constructed on the memory hierarchy. 2) Load the data to be processed. The data to be processed can be stored in the non-volatile memory, and the computing node loads the data to be processed into the local memory of the computing node. For example, the computing node loads the data to be processed from the non-volatile memory pool into the local memory according to the memory operation instruction, which eliminates the access overhead of the file system stack. The computing node can also acquire the data to be processed stored in the remote node by using the remote direct memory access (RDMA), so as to realize efficient access and reduce the CPU overhead and network protocol stack overhead of remote data access.

[0105] ​2) Data processing stage: 3o, the data processing request is used to process the data to be processed using the memory operation instruction to obtain the processed data, for example, each computing node sorts the processed data to obtain the intermediate data. The computing node selects the storage medium for storing the processed data from the global memory pool according to the data volume of the processed data, for example, the processed data is stored in the HBM, DRAM or SCM. When the storage capacity in the global memory pool is insufficient, it can be solved by remote memory expansion or memory disk expansion, that is, the processed data is stored in other storage media. The destination node reads the intermediate data from the source node or writes the intermediate data from the source node to the destination node to complete the data exchange (shuffle). If the global memory pool is built by independent storage media (such as the storage media of the computing node and the storage media of the storage node), the source node can write the intermediate data to the global memory pool, and then the destination node reads the intermediate data from the global memory pool. 4o, the sorting result is merged.

[0106] 3) Persistent stage: o5, test report and merged sorting result are written to local non-volatile memory or local memory of global memory pool, and are asynchronously written to storage space of storage node by global memory pool component.

[0107] 4) Cleaning stage: o6, local memory or global memory pool is released.

[0108] It can be understood that, in order to realize the functions in the above embodiments, the computing device includes the hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in combination with the embodiments disclosed in the present application, the present application can be realized in the form of hardware or hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application scenario and design constraints of the technical solution.

[0109] The data processing method of the fusion system provided by the present embodiment is described in detail above. Figures 1 to 7 The data processing device provided by the present embodiment will be described below in combination with Figure 8 .

[0110] Figure 8 The structure schematic diagram of the possible data processing device provided by the present embodiment. These data processing devices can be used to realize the functions of the computing device or the computing node in the above method embodiments, and thus can also realize the beneficial effects possessed by the above method embodiments. In the present embodiment, the data processing device can be a computing node 121 as shown in Figure 5 , and can also be a module (such as a chip) applied to a server.

[0111] As shown in Figure 8As shown, the data processing apparatus 800 comprises a communication module 810, a data processing module 820 and a storage module 830. The data processing apparatus 800 is configured to implement the functions of the computing node 121 in the method embodiments shown in the above Figure 5

[0112] The communication module 810 is configured to obtain the processing request data.

[0113] The data processing module 820 is configured to perform the memory operation on the global memory pool according to the memory operation instruction, so as to process the processing request data. For example, the data processing module 820 is configured to perform steps 510 and 520 in the method embodiment shown in the above Figure 5

[0114] The storage module 830 is configured to store the memory operation instruction, so as to facilitate the data processing module 820 to obtain the processed data. For example, the communication module 810 is configured to perform S560 in the method embodiment shown in the above Figure 5

[0115] The data processing module 820 is specifically configured to read the to-be-processed data and the application data from the global memory pool; perform processing on the to-be-processed data according to the memory operation instruction to obtain the processed data, and write the processed data into the storage space indicated by the first address in the global memory pool.

[0116] The data processing module 820 is further configured to read the processed data from the global memory pool according to the first address.

[0117] The data processing module 820 is further configured to pre-fetch data from the storage node according to the memory operation instruction and store the data into the global memory pool.

[0118] The data processing module 820 is further configured to perform the memory operation on the data between the global memory pool and the storage node according to the data cold and hot characteristics and based on the memory operation instruction.

[0119] It should be understood that the data processing apparatus 800 of the embodiments of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), which can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The data processing apparatus 800 can also be implemented by software Figure 5 or​​​Figure 6 The data processing method shown and each module thereof can also be a software module, and the data processing apparatus 800 and each module thereof can also be a software module.

[0120] The data processing apparatus 800 according to the embodiments of the present application can correspond to performing the method described in the embodiments of the present application, and the above and other operations and / or functions of each unit in the data processing apparatus 800 are respectively implemented to achieve Figure 5 or Figure 6 the corresponding flow of each method in the embodiments of the present application, for brevity, will not be repeated here.

[0121] Figure 9 A structural schematic diagram of a computing device 900 provided in the embodiments is shown. As shown in the figure, the computing device 900 includes a processor 910, a bus 920, a memory 930, a communication interface 940, and a memory unit 950 (which can also be referred to as a main memory unit). The processor 910, the memory 930, the memory unit 950, and the communication interface 940 are connected through the bus 920.

[0122] It should be understood that in the embodiments, the processor 910 can be a CPU, and the processor 910 can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0123] The processor can also be a graphics processor (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling program execution of the solutions of the present application.

[0124] The communication interface 940 is configured to realize communication between the computing device 900 and external devices or devices. In the embodiments, the computing device 900 is configured to realize the functions of the computing node 121 shown, and the communication interface 940 is configured to obtain processing request data, and the computing node 121 performs memory operations on the global memory pool according to the memory operation instructions. Figure 5

[0125] ​The bus 920 can include a pathway for conveying information among the aforementioned components (e.g., the processor 910, the memory unit 950, and the storage 930). The bus 920 can include, in addition to a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, the various buses are labeled as the bus 920 in the figure. The bus 920 can be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 920 can be divided into an address bus, a data bus, a control bus, etc.

[0126] As an example, the computing device 900 can include multiple processors. The processors can be a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). In the present embodiment, the computing device 900 is configured to implement the functions of the computing node 121 shown. Figure 5 When the processor 910 is configured to implement the functions of the computing node 121 shown, the processor 910 can perform the memory operation of the processing request data on the global memory pool according to the memory operation instruction.

[0127] It is worth noting that, Figure 9 In the present embodiment, only the computing device 900 includes one processor 910 and one storage 930 as an example. Here, the processor 910 and the storage 930 are respectively used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0128] The memory unit 950 can correspond to the global memory pool in the above method embodiments for storing processed data and other information. The memory unit 950 can be a volatile memory pool or a non-volatile memory pool, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0129] The memory 930 can correspond to the storage medium in the above method embodiments for storing computer instructions, memory operation instructions, storage strategies, and other information, for example, a magnetic disk such as a mechanical hard disk or a solid-state hard disk.

[0130] The above computing device 900 can be a general-purpose device or a special-purpose device. For example, the computing device 900 can be an edge device (e.g., a box carrying a processing-capable chip) and the like. Alternatively, the computing device 900 can also be a server or other computing-capable device.

[0131] It should be understood that the computing device 900 according to the present embodiment can correspond to the data processing apparatus 800 in the present embodiment, and can correspond to the respective subject performing the method in any one of Figure 5 or Figure 6 The above and other operations and / or functions of the respective modules in the data processing apparatus 800 are respectively for implementing the respective flows of the methods in Figure 5 or Figure 6 For brevity, they will not be described here.

[0132] The steps of a method or algorithm described in connection with the present disclosure can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, a removable media, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. Alternatively, the processor and the storage medium can reside as discrete components in a computing device. Moreover, in some embodiments, the processor and the storage medium can reside as discrete components in a computing device.

[0133] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; or an optical medium, for example, a digital video disc (digital video disc, DVD); or a semiconductor medium, for example, a solid state disk (solid state drive, SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method for a fusion system, characterized in that, The fusion system includes compute nodes and storage nodes. The compute nodes are connected to the storage nodes via a network, and the storage media of the compute nodes and the storage media of the storage nodes constitute a global memory pool. The method includes: Acquire processing request data, which is used to indicate the operation to be performed on the data to be processed, and the processing request data includes user requirements; Obtain memory operation instructions based on the processing request data; According to the memory operation instructions, memory operations are performed on the processing request data in the global memory pool. When the object of the memory operation is the storage medium of a storage node in the global memory pool, the memory operation is performed on the storage medium of the storage node based on the same memory interface as the storage medium of the computing node. The memory operation includes: determining the storage location of the processed data in the global memory pool according to the user requirements and storage medium characteristics. The user requirements are used to indicate requirements related to the storage medium characteristics. The processed data is obtained by processing the data to be processed using the memory operation instructions according to the processing request data.

2. The method according to claim 1, characterized in that, Memory operation instructions include at least one of memory allocation, memory setting, memory copying, memory moving, memory release, and memory comparison.

3. The method according to claim 2, characterized in that, The storage media of the global memory pool include memory, memory servers, and storage-class memory (SCM).

4. The method according to any one of claims 1-3, characterized in that, Perform memory operations on the global memory pool according to memory operation instructions, including: Read the data to be processed from the global memory pool; Based on the processing request data, the memory operation instructions are used to process the data to be processed to obtain processed data, and the processed data is written into the storage space indicated by the first address in the global memory pool.

5. The method according to claim 4, characterized in that, Writing the processed data to the storage space indicated by the first address in the global memory pool includes: Based on the user requirements and the characteristics of the storage medium, the storage space indicated by the first address in the global memory pool is determined to be where the processed data will be written. The characteristics of the storage medium include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, CPU consumption, energy efficiency ratio, and reliability.

6. The method according to claim 4, characterized in that, The storage space indicated by the first address includes one of the storage space provided by the storage medium of the computing node and the storage space provided by the storage medium of the storage node.

7. The method according to claim 4, characterized in that, After writing the processed data to the storage space indicated by the first address in the global memory pool, the method further includes: The processed data is read from the global memory pool according to the first address.

8. The method according to claim 7, characterized in that, Reading the processed data from the global memory pool according to the first address includes: The processed data is read from the global memory pool according to the first address and written to the storage node.

9. The method according to claim 1, characterized in that, The method further includes: Data is prefetched from the storage node according to the memory operation instructions and stored in the global memory pool.

10. The method according to claim 1, characterized in that, The method further includes: Based on the hot and cold characteristics of the data, memory operations are performed between the global memory pool and the storage node according to the memory operation instructions.

11. A data processing device for a fusion system, characterized in that, The fusion system includes compute nodes and storage nodes. The compute nodes are connected to the storage nodes via a network, and the storage media of the compute nodes and the storage media of the storage nodes constitute a global memory pool. The device includes: The communication module is used to acquire processing request data, which is used to indicate the operation to be performed on the data to be processed, and the processing request data includes user requirements; A data processing module is configured to obtain memory operation instructions based on the processing request data, and perform memory operations on the processing request data in the global memory pool according to the memory operation instructions. When the object of the memory operation is the storage medium of a storage node in the global memory pool, the memory operation is performed on the storage medium of the storage node using the same memory interface as the memory medium accessed by the computing node. The memory operation includes: determining the storage location of the processed data in the global memory pool based on the user requirements and storage medium characteristics, wherein the user requirements indicate requirements related to the storage medium characteristics; and the processed data is obtained by processing the data to be processed using the memory operation instructions based on the processing request data.

12. The apparatus according to claim 11, characterized in that, Memory operation instructions include at least one of memory allocation, memory setting, memory copying, memory moving, memory release, and memory comparison.

13. The apparatus according to claim 12, characterized in that, The storage media of the global memory pool include memory, memory servers, and storage-class memory (SCM).

14. The apparatus according to any one of claims 11-13, characterized in that, When the data processing module performs memory operations on the global memory pool to process the requested data according to memory operation instructions, it is specifically used for: Read the data to be processed from the global memory pool; Based on the processing request data, the memory operation instructions are used to process the data to be processed to obtain processed data, and the processed data is written into the storage space indicated by the first address in the global memory pool.

15. The apparatus according to claim 14, characterized in that, When the data processing module writes the processed data into the storage space indicated by the first address in the global memory pool, it is specifically used for: Based on the user requirements and the characteristics of the storage medium, the storage space indicated by the first address in the global memory pool is determined to be where the processed data will be written. The characteristics of the storage medium include at least one of write latency, read latency, total storage capacity, available storage capacity, access speed, CPU consumption, energy efficiency ratio, and reliability.

16. The apparatus according to claim 14, characterized in that, The storage space indicated by the first address includes one of the storage space provided by the storage medium of the computing node and the storage space provided by the storage medium of the storage node.

17. The apparatus according to claim 14, characterized in that, The data processing module is further configured to read the processed data from the global memory pool according to the first address.

18. The apparatus according to claim 17, characterized in that, When the data processing module reads the processed data from the global memory pool according to the first address, it is specifically used for: The processed data is read from the global memory pool according to the first address and written to the storage node.

19. The apparatus according to claim 11, characterized in that, The data processing module is also used to prefetch data from the storage node according to the memory operation instructions and store it in the global memory pool.

20. The apparatus according to claim 11, characterized in that, The data processing module is also used to perform memory operations on data between the global memory pool and the storage node based on the memory operation instructions, according to the data's hot / cold characteristics.

21. A computing device, characterized in that, The computing device includes a memory and at least one processor, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, it performs the operational steps of the method according to any one of claims 1-10.

22. A fusion system, characterized in that, The fusion system includes storage nodes and computing nodes. The computing nodes are connected to the storage nodes via a network. The storage media of the computing nodes and the storage media of the storage nodes are uniformly addressed to form a global memory pool. The computing nodes are used to execute the operation steps of the method according to any one of claims 1-10.

Images