A data storage method and related device

Abstract

The application discloses a data storage method and related equipment. In the application, after a network device obtains target data required to be stored by a target application, the network device can store the target data in a target memory pool according to a type of the target data. The target memory pool includes a local memory area and a shared memory area. The local memory area is used to indicate a local memory resource of the network device, and the shared memory area is used to indicate a logical memory formed by memory resources of the network device and / or a plurality of other network devices. The network device can store the target data into different memory areas in the target memory pool according to the type of the target data, thereby improving the storage efficiency of the target data.

Description

[0001] This application claims priority to Chinese Patent Application No. 202110713213.9, filed on June 25, 2021, entitled "A Memory System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communications, and more particularly to a data storage method and related equipment. Background Technology

[0003] Currently, computing power pooling and storage pooling are widely used as key infrastructure technologies. To improve application performance, memory pooling is gaining attention in the industry. In traditional memory pooling solutions, the approach typically starts at the architecture level. When network devices determine that local memory is insufficient or data sharing and exchange are necessary, they can access remote memory to achieve global memory sharing and pooling.

[0004] Traditional pooling schemes achieve remote memory sharing and pooling at the system architecture level. Essentially, they extend the local memory of network devices, providing efficient and transparent global memory pool services for network devices running standalone applications. However, for multiple network devices running non-standalone applications, traditional pooling schemes cannot effectively determine the number of network devices and memory capacity required after partitioning (SWAP). Furthermore, all data is indiscriminately stored in the same memory region, resulting in low storage efficiency. Summary of the Invention

[0005] This application provides a data storage method in which a network device can store target data into different memory regions in a target memory pool according to the type of target data, thereby improving the storage efficiency of target data.

[0006] The first aspect of this application provides a data storage method, the method comprising: a network device acquiring target data, the network device running a target application, the target data being used to indicate data that the target application needs to store; the network device storing the target data in a target memory pool according to the type of the target data, the target memory pool including a local memory region and a shared memory region, the local memory region being used to indicate the local memory resources of the network device, and the shared memory region being used to indicate logical memory composed of the memory resources of the network device and / or multiple other network devices.

[0007] In this application, after the network device obtains the target data that the target application needs to store, the network device can store the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory region and a shared memory region. The local memory region indicates the network device's local memory resources, and the shared memory region indicates the logical memory constituted by the memory resources of the network device and / or multiple other network devices. The network device can store the target data in different memory regions within the target memory pool according to the type of target data, thereby improving the storage efficiency of the target data.

[0008] In one possible implementation of the first aspect, the type of the target data includes non-shared data, which is used to indicate data that does not require access by other network devices. The network device stores the target data in a target memory pool according to the type of the target data, including: the network device confirming that the target data is the non-shared data; and the network device storing the non-shared data in a local memory area according to the type of the non-shared data.

[0009] In this possible implementation, the network device determines whether the target data is shared data by checking with the target application to see if the target data needs to be exchanged or shared with other network devices. If the target application determines that the target data does not need to be exchanged or shared with other network devices, then the target application can confirm that the target data is non-shared data. The network device can distinguish and store non-shared data in different locations within its local memory area according to the type of non-shared data.

[0010] In one possible implementation of the first aspect, the type of non-shared data includes cold data, which is used to indicate data that is accessed infrequently by the network device. The network device stores the non-shared data in the local memory area according to the type of the non-shared data, including: the network device confirming that the non-shared data is the cold data; and the network device storing the cold data to the disk of the local memory area.

[0011] In this possible implementation, the network device determines whether the target data is cold data by checking whether the access frequency of the target data reaches a preset threshold through the target application. If the target application confirms that the access frequency of the target data does not reach the preset threshold, then the target application can confirm that the target data is cold data. Since cold data is accessed less frequently than hot data is accessed more frequently, cold data can be preferentially stored in memory resources with slower response times, such as disks. Optionally, the disk medium can be a hard disk drive (HDD), a solid state storage device (SSD), or other types of media; specific details are not limited here.

[0012] In one possible implementation of the first aspect, the type of non-shared data includes hot data, which is used to indicate data that the network device accesses frequently. The network device stores the non-shared data in the local memory area according to the type of the non-shared data, including: the network device confirming that the non-shared data is the hot data; and the network device storing the hot data in the cache of the local memory area.

[0013] In this possible implementation, the network device determines whether the target data is hot data by checking whether the access frequency of the target data reaches a preset threshold. If the target application confirms that the access frequency of the target data reaches the preset threshold, then the target application can confirm that the target data is hot data. Since cold data is accessed less frequently and hot data is accessed more frequently, hot data can be preferentially stored in memory resources with faster response speeds, such as caches, to further save the time consumed by the network device in reading hot data. Optionally, the cache medium can be Dynamic Random Access Memory (DRAM), or other types of media, which are not limited here.

[0014] In one possible implementation of the first aspect, the type of the target data includes shared data, which is used to indicate data that needs to be accessed by other network devices. The network device stores the target data in a target memory pool according to the type of the target data, including: the network device confirming that the target data is the shared data; and the network device storing the shared data in the shared memory region according to the type of the shared data.

[0015] In this possible implementation, after the network device confirms that the target data is shared data, it can distinguish and store the shared data at different locations within the shared memory region based on the type of shared data. The shared data types include at least persistent and non-persistent data. Persistent data indicates data that the network device needs to read and use multiple times, while non-persistent data indicates data that the network device will discard after a few reads. By storing shared data within the shared memory region, other network devices can directly retrieve the shared data from the shared memory region via Remote Direct Memory Access (RDMA) and / or Data Streaming Assessment (DSA) protocols, reducing the time spent retrieving the target data and improving operational efficiency.

[0016] In one possible implementation of the first aspect, the type of shared data includes persistent data, and the network device stores the shared data in the shared memory region according to the type of shared data, including: the network device confirming that the shared data is persistent data; and the network device storing the persistent data in a non-volatile medium of the shared memory region.

[0017] In this possible implementation, the network device confirms whether the target data is persistent through the target application, thus confirming that the target data is persistent. Optionally, the network device can also confirm whether the target data is persistent through other methods, which are not limited here. Since persistent data needs to be read multiple times, it needs to be stored in memory for a long time. The network device can preferentially store persistent data in non-volatile media to prevent persistent data loss. Optionally, the non-volatile media can be phase-change memory (PCM), or other types of media, which are not limited here.

[0018] In one possible implementation of the first aspect, the type of shared data includes non-persistent data, and the network device stores the shared data in the shared memory region according to the type of shared data, including: the network device confirming that the shared data is the non-persistent data; and the network device storing the non-persistent data in the volatile medium of the shared memory region.

[0019] In this possible implementation, the network device determines whether the target data is non-persistent through the target application. Since non-persistent data does not need to be read multiple times, it does not need to be stored in memory for a long time, and the network device can preferentially store non-persistent data in volatile media. Optionally, the volatile media can be DRAM, or other types of media, which are not limited here.

[0020] In one possible implementation of the first aspect, the target memory pool includes a northbound interface, which includes a first-level interface, a second-level interface, and a third-level interface. The first-level interface includes a memory semantic interface, the second-level interface includes a distributed data structure interface, and the third-level interface includes an application semantic interface, a file semantic interface, and / or a programming model interface.

[0021] In this possible implementation, the target memory pool can also include multiple northbound interfaces, allowing different developers to use the target memory pool through appropriate interfaces based on type and performance requirements.

[0022] In one possible implementation of the first aspect, the target application schedules the third-level interface by selecting a specific interface object from the third-level interface, the third-level interface schedules the second-level interface by selecting a specific interface object from the second-level interface, the second-level interface schedules the first-level interface by selecting a specific interface object from the first-level interface, and the first-level interface is used to schedule the target memory pool.

[0023] In one possible implementation of the first aspect, the memory semantic interface includes BigMemory, persistent multiple copies (plog), and / or memory objects (Memobj).

[0024] In one possible implementation of the first aspect, the distributed data structure interface includes key-value (KV), hash, B+ tree, array, and / or matrix.

[0025] In one possible implementation of the first aspect, the application semantic interface includes Shuffle, Cache, Write-Ahead Logging (WAL), Message Queue (MQ), and / or AI Parameter Servers; the file semantic interface includes Portable Operating System Interface (POSIX), Hadoop Distributed File System (HDFS), MPI-IO, and / or Java File; and the programming semantic interface includes UPC, UPC++, OpenSHMEM, and / or X10.

[0026] In one possible implementation of the first aspect, the target memory pool includes the local memory region and the shared memory region, the local memory region including a disk and a cache, and the shared memory region including volatile media and non-volatile media.

[0027] A second aspect of this application provides a network device including at least one processor, a memory, and a communication interface. The processor is coupled to the memory and the communication interface. The memory stores instructions, the processor executes the instructions, and the communication interface communicates with other network devices under the control of the processor. When executed by the processor, the instructions cause the network device to perform the methods described in the first aspect or any possible implementation thereof.

[0028] A third aspect of this application provides a computer-readable storage medium storing a program that causes the network device to perform the methods described in the first aspect or any possible implementation thereof.

[0029] The fourth aspect of this application provides a computer program product storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by the processor, the processor executes the method described in the first aspect or any possible implementation thereof.

[0030] The fifth aspect of this application provides a chip including a processor and a communication interface, wherein the processor is coupled to the communication interface, and the processor is used to read instructions to execute the method described in the first aspect or any possible implementation thereof.

[0031] The sixth aspect of this application is a communication system, which includes the network device described in the first aspect or any possible implementation thereof.

[0032] A seventh aspect of this application provides a converged memory system, the converged memory system including multiple network devices, the network devices using a target memory pool to store data;

[0033] The target memory pool includes a local memory region and a shared memory region. The local memory region includes a disk and a cache, and the shared memory region includes volatile media and non-volatile media.

[0034] An eighth aspect of this application provides a converged memory system, the converged memory system including a first network device and a plurality of second network devices, wherein the first network device uses a target memory pool to store data;

[0035] The target memory pool includes a local memory region and a shared memory region. The local memory region is used to indicate the local memory of the first network device and includes disks and caches. The shared memory region is used to indicate the logical memory constructed after pooling memory resources in multiple second network devices and includes volatile media and non-volatile media.

[0036] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0037] In this application, after the network device obtains the target data that the target application needs to store, the network device can store the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory region and a shared memory region. The local memory region indicates the network device's local memory resources, and the shared memory region indicates the logical memory constituted by the memory resources of the network device and / or multiple other network devices. The network device can store the target data in different memory regions within the target memory pool according to the characteristics of the target data, thereby improving the storage efficiency of the target data. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating an application scenario of a communication system provided in this application;

[0039] Figure 2 This is a schematic diagram illustrating an application scenario of another communication system provided in this application;

[0040] Figure 3 This is a schematic diagram illustrating an application of a data storage method provided in this application;

[0041] Figure 4 A schematic diagram of the structure of a network device provided in this application;

[0042] Figure 5 A schematic diagram of a northbound interface provided in this application;

[0043] Figure 6 This application provides an application diagram of a northbound interface;

[0044] Figure 7 Another structural schematic diagram of a network device provided in this application;

[0045] Figure 8 This is a schematic diagram of another structure of a network device provided in this application. Detailed Implementation

[0046] The embodiments of this application will now be described with reference to the accompanying drawings. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0047] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0048] In this application, "and / or" is merely a descriptive term for the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0049] Currently, computing power pooling and storage pooling are widely used as key infrastructure technologies. To improve application performance, memory pooling is gaining attention in the industry. In traditional memory pooling solutions, the approach typically starts at the architecture level. When network devices determine that local memory is insufficient or data sharing and exchange are necessary, they can access remote memory to achieve global memory sharing and pooling.

[0050] Traditional pooling schemes achieve remote memory sharing and pooling at the system architecture level. Essentially, they extend the local memory of network devices, providing efficient and transparent global memory pool services for network devices running standalone applications. However, for multiple network devices running non-standalone applications, traditional pooling schemes cannot effectively determine the number of network devices and memory capacity required after partitioning (SWAP). Furthermore, all data is indiscriminately stored in the same memory region, resulting in low storage efficiency.

[0051] In response to the traditional pooling schemes described in the above method examples, this application provides a data storage method, a communication system, and a network device. The network device can store target data into different memory regions within the target memory pool based on the characteristics of the target data, thereby improving the storage efficiency of the target data.

[0052] The following examples, in conjunction with the accompanying drawings, will illustrate the data storage method, communication system, and network device provided in this application. First, the communication system provided in this application will be introduced.

[0053] The communication system provided in this application can also be called a converged memory system (large memory system). The converged memory system can be deployed in two ways, which are described in detail below.

[0054] Method 1: Integrated deployment.

[0055] Figure 1 This is a schematic diagram of an application scenario for a communication system provided in this application.

[0056] In this application, it is assumed that the network devices in the converged memory system are compute nodes. The target memory pool in a compute node includes a local memory region and a shared memory region. The local memory region indicates the local memory resources of the compute node, and the shared memory region indicates the logical memory composed of the memory resources of the network device and multiple other network devices. If the converged memory system is deployed using a converged deployment approach, all memory resources in the shared memory region of the target memory pool are provided by the compute nodes in the compute node cluster, and some or all nodes in the compute node cluster contribute a portion of their local memory as the shared memory region.

[0057] Method 2: Separate deployment.

[0058] Figure 2 This is a schematic diagram of another application scenario of the communication system provided in this application.

[0059] In this application, it is assumed that the network devices in the converged memory system are compute nodes. The target memory pool in the compute node includes a local memory region and a shared memory region. The local memory region indicates the memory resources local to the compute node, and the shared memory region indicates the logical memory composed of the memory resources of multiple other network devices. If the converged memory system is deployed in a separate deployment manner, the memory resources in the shared memory region of the target memory pool are all provided by storage nodes in a separate memory cluster. The storage nodes are only used to provide memory resources in the shared memory region and do not handle other computing services.

[0060] In this application, the converged memory system can be deployed in a converged deployment manner, or in a separate deployment manner, or in other ways, without being limited here.

[0061] The examples above illustrate two deployment methods for converged memory systems. The examples below illustrate the specific implementation of the network device in the converged memory system provided in this application.

[0062] In this application, optionally, the network device can be a computing node, and the converged system can include one or more computing nodes. Multiple computing nodes can form a computing node cluster, and the computing nodes can be interconnected. The computing node can be a server, desktop computer, or the controller, hard disk enclosure, etc. of a storage array.

[0063] Functionally, computing nodes are primarily used for data computation and processing. Hardware-wise, a computing node includes at least a processor, memory, and a control unit. The processor, or central processing unit (CPU), processes data from outside the network device or data generated internally within the computing node. Memory refers to the device used to store data; it can be RAM or a hard disk. RAM is internal storage that directly exchanges data with the processor; it can read and write data at any time and is very fast, serving as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory; for example, RAM can be random access memory (RAM) or read-only memory (ROM). For instance, RAM can be dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a semiconductor memory and, like most random access memory (RAM), is a type of volatile memory. SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and memory. Storage-class memory can provide faster read and write speeds than hard drives, but its access speed is slower than DRAM, and it is also cheaper than DRAM.

[0064] The above example illustrates the structure of the communication system (converged memory system) provided in this application. The following example will describe the data storage method provided in this application in detail with reference to the communication system described in the above example.

[0065] Figure 3 This is a schematic diagram illustrating an application of a data storage method provided in this application.

[0066] In this application, please refer to Figure 3 ,like Figure 3 As shown, the data storage method provided in this application includes at least steps 201 to 202.

[0067] 201. Network devices acquire target data.

[0068] In this application, a target application runs on a network device, and target data is used to indicate the data that the target application needs to store. Optionally, the target application running on the network device can be an application with large data types, a high-performance computing (HPC) cluster application, an artificial intelligence (AI) application, a database application, a World Wide Web (Web) application, or other types of applications; no specific limitations are made here.

[0069] In this application, the target data may be data that a certain process of the target application needs to store. This process may be a local process of a single-process application or a local process of a distributed application. The specific process is not limited here.

[0070] 202. The network device stores the target data in the target memory pool according to the type of the target data.

[0071] In this application, the target memory pool includes a local memory region and a shared memory region. The local memory region is used to indicate the local memory resources of the network device, and the shared memory region is used to indicate the logical memory composed of the memory resources of the network device and / or multiple other network devices.

[0072] In this application, after the network device obtains the target data that the target application needs to store, the network device can store the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory region and a shared memory region. The local memory region indicates the network device's local memory resources, and the shared memory region indicates the logical memory constituted by the memory resources of the network device and / or multiple other network devices. The network device can store the target data in different memory regions within the target memory pool according to the type of target data, thereby improving the storage efficiency of the target data.

[0073] In this application, steps 201 to 202 in the above method examples illustrate the data storage method provided by this application. In step 202 of the above method examples, the network device stores the target data in the target memory pool according to the type of the target data. This specific implementation method will be described in the following method examples.

[0074] In this application, the type of target data may include non-shared data and shared data. Non-shared data is used to indicate data that does not require access from other network devices, while shared data is used to indicate data that requires access from other network devices.

[0075] Scenario 1: The target data is non-shared data.

[0076] The network device confirmed that the target data was non-shared data.

[0077] In this application, the network device determines whether the target data is shared data by confirming with the target application whether the target data needs to be exchanged or shared with other network devices. If the target application confirms that the target data does not need to be exchanged or shared with other network devices, then the target application can confirm that the target data is non-shared data.

[0078] Optionally, network devices can verify whether target data is non-shared data through the target application, or they can verify whether target data is non-shared data through other means besides the target application; no specific limitation is made here.

[0079] Network devices store non-shared data in their local memory areas according to the type of non-shared data.

[0080] In this application, after the network device confirms that the target data is non-shared data, the network device can distinguish and store the non-shared data in different locations within its local memory area according to the type of non-shared data. The types of non-shared data include at least cold data and hot data, with cold data indicating data that is accessed infrequently by the network device and hot data indicating data that is accessed frequently by the network device.

[0081] Scenario 1: Non-shared data is cold data.

[0082] The network device identifies the non-shared data as cold data.

[0083] Optionally, in this application, the network device determines whether the target data is cold data by checking whether the access frequency of the target data reaches a preset threshold through the target application. If the target application confirms that the access frequency of the target data does not reach the preset threshold, then the target application can confirm that the target data is cold data. Optionally, the network device may also confirm whether the target data is cold data through other methods, which are not limited here.

[0084] Network devices store cold data in the local memory area of ​​the disk.

[0085] In this application, since cold data is accessed less frequently and hot data is accessed more frequently, cold data can be preferentially stored in memory resources with slower response times, such as disks. Optionally, the disk medium can be an HDD, an SSD, or other types of media; no specific limitations are made here.

[0086] Scenario 2: Non-shared data is hot data.

[0087] The network device identifies non-shared data as hot data.

[0088] Optionally, in this application, the network device determines whether the target data is hot data by checking whether the access frequency of the target data reaches a preset threshold through the target application. If the target application confirms that the access frequency of the target data reaches the preset threshold, then the target application can confirm that the target data is hot data. Optionally, the network device may also confirm whether the target data is hot data through other methods, which are not limited here.

[0089] Network devices store hot data in a cache in the local memory area.

[0090] In this application, since cold data is accessed less frequently and hot data is accessed more frequently, hot data can be preferentially stored in memory resources with faster response times, such as cache. Optionally, the cache medium can be DRAM, or other types of media; no specific limitation is made here.

[0091] Scenario 2: The target data is shared data.

[0092] The network device confirmed that the target data was shared data.

[0093] In this application, the network device determines whether the target data is shared data by confirming with the target application whether the target data needs to be exchanged or shared with other network devices. If the target application confirms that the target data needs to be exchanged or shared with other network devices, then the target application can confirm that the target data is shared data.

[0094] Optionally, network devices can determine whether target data is shared data through the target application, or they can determine whether target data is shared data through other means besides the target application; no specific limitation is made here.

[0095] Network devices store shared data in a shared memory area according to the type of shared data.

[0096] In this application, after the network device confirms that the target data is shared data, the network device can distinguish and store the shared data at different locations within the shared memory region according to the type of shared data. The types of shared data include at least persistent data and non-persistent data. Persistent data is used to indicate data that the network device needs to read and use multiple times, while non-persistent data is used to indicate data that the network device will discard after reading a few times.

[0097] Scenario 1: The shared data is persistent data.

[0098] The network device confirms that the shared data is persistent.

[0099] In this application, optionally, the network device confirms whether the target data is persistent data through the target application; optionally, the network device may also confirm whether the target data is persistent data through other means, which are not limited here.

[0100] Network devices store persistent data in non-volatile media within a shared memory region.

[0101] In this application, since persistent data needs to be read multiple times, it needs to be stored in memory for a long time. Network devices can preferentially store persistent data in non-volatile media to prevent data loss. Optionally, the non-volatile media can be PCM, or other types of media; no specific limitation is made here.

[0102] Scenario 2: The shared data is non-persistent.

[0103] The network device confirms that the shared data is persistent.

[0104] Optionally, in this application, the network device may determine whether the target data is non-persistent data through the target application. Optionally, the network device may also determine whether the target data is non-persistent data through other means, which are not limited here.

[0105] Network devices store non-persistent data in volatile media within a shared memory region.

[0106] In this application, since non-persistent data does not need to be read multiple times, it does not need to be stored in memory for a long time. Network devices can preferentially store non-persistent data in volatile media. Optionally, the volatile media can be DRAM, or other types of media, which are not limited here.

[0107] Figure 4 This is a schematic diagram of the structure of a network device provided in this application.

[0108] The following is based on Figure 4 This example illustrates the process by which a network device stores target data in different areas of a target memory pool based on the type of target data.

[0109] Please see Figure 4 Assuming node 1 is a network device, the target memory pool on node 1 includes a local memory region and a shared memory region. It is a memory pool formed by the fusion of single-node multi-level memory (local memory region) and distributed memory pool (shared memory region). In the distributed memory pool, volatile and non-volatile media can be distributed independently.

[0110] In this application, the local memory region has multiple layers, including cache and disk. Various media types, such as DRAM, PCM, and SSD, constitute different layers within the local memory region, thereby enabling multi-level memory expansion using DRAM, PCM, and SSD. The shared memory region includes a global memory pool composed of volatile media and a memory pool composed of non-volatile media (Persistence Memory Pool). The global memory pool composed of volatile media provides globally accessible memory space, while the global memory pool composed of non-volatile media provides globally persistent memory space.

[0111] If the target application (app) needs to store target data, the target application first determines the characteristics of the target data. If the target data is non-shared data, it determines the characteristics of non-shared data. If it is cold data, it is stored on disk; if it is hot data, it is stored in cache. If the target data is shared data, if it is persistent data, it is stored in a global memory pool composed of non-volatile media; if it is non-persistent data, it is stored in a global memory pool composed of volatile media.

[0112] The above method example illustrates the process by which the network device stores target data in the target memory pool according to the type of target data in this application. The target memory pool provided in this application may also include a variety of northbound interfaces, and different developers can use the target memory pool through appropriate interfaces according to the type and performance requirements.

[0113] Figure 5 This is a schematic diagram of a northbound interface provided in this application.

[0114] Please see Figure 5 The target memory pool includes a northbound interface, which includes a first-level interface, a second-level interface, and a third-level interface. The first-level interface includes a memory semantic interface, the second-level interface includes a distributed data structure interface, and the third-level interface includes an application semantic interface, a file semantic interface, and / or a programming model interface.

[0115] In this application, optionally, the memory semantic interface includes Big Memory, persistent multiple copies (plog), memory objects (Memobj), and other interfaces, which are not limited here.

[0116] In this application, optionally, the distributed data structure interface includes key-value (KV), the distributed data interface may include hash, the distributed data interface may include B+ tree, the distributed data interface may include array, the distributed data interface may include matrix, and the distributed data interface may also include other interfaces, which are not limited here.

[0117] In this application, optionally, the application semantic interface may include Shuffle, Cache, WAL, Message Queue (MQ), AI Parameter Servers, and other interfaces, which are not limited here.

[0118] In this application, optionally, the file semantic interface may include a Portable Operating System Interface (Posix), a Hadoop Distributed File System (HDFS), MPI-IO, and / or a Java File.

[0119] In this application, the programming semantic interface may optionally include UPC, UPC++, OpenSHMEM, X10, or other interfaces, which are not specifically limited here.

[0120] An interface object is an element in an interface. For example, Big Memory mentioned above can be understood as a specific interface object in the memory semantic interface.

[0121] Figure 6 This is a schematic diagram illustrating the application of a northbound interface provided in this application.

[0122] For example, let's take AI as the target application to illustrate the process of using the target memory pool through the northbound interface.

[0123] Optionally, the northbound interface may also include Figure 6 The set communication semantic interface and memory pool operator interface are shown.

[0124] In this application, the AI ​​application schedules the gather interface through the PS interface. The gather interface is used to query target data. The gather interface selects the target matrix for data retrieval through the matrix selection interface. After selecting the Big Memory interface, the matrix inputs the target matrix into the target memory pool to find the target data. After the target data is found through the Big Memory interface, it is returned to scatter through the Big Memory interface and the matrix interface. The scatter interface then uses the target data to update the data in the AI ​​application via the PS interface.

[0125] In this application, after the network device obtains the target data that the target application needs to store, the network device can store the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory region and a shared memory region. The local memory region indicates the network device's local memory resources, and the shared memory region indicates the logical memory constituted by the memory resources of the network device and / or multiple other network devices. The network device can store the target data in different memory regions within the target memory pool according to the type of target data, thereby improving the storage efficiency of the target data.

[0126] The above examples provide a different implementation of a data storage method. Below is a network device 30, such as... Figure 7 As shown, the network device 30 is used to execute the steps performed by the network device (compute node) in the above example. For details on these execution steps and their corresponding beneficial effects, please refer to the corresponding examples above; they will not be repeated here. The network device 30 includes:

[0127] Acquisition unit 301 is used to acquire target data, wherein a target application is running on the network device, and the target data is used to indicate the data that the target application needs to store.

[0128] Storage unit 302 is used to store the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory area and a shared memory area. The local memory area is used to indicate the local memory resources of the network device, and the shared memory area is used to indicate the logical memory composed of the memory resources of the network device and / or multiple other network devices.

[0129] In one possible implementation, the type of the target data includes non-shared data, which is used to indicate data that does not require access from other network devices.

[0130] Confirmation unit 303 is used to confirm that the target data is the non-shared data;

[0131] The storage unit 302 is used to store the non-shared data in the local memory area according to the type of the non-shared data.

[0132] In one possible implementation, the type of non-shared data includes cold data, which is used to indicate data that the network device accesses infrequently.

[0133] The confirmation unit 303 is used to confirm that the non-shared data is the cold data;

[0134] The storage unit 302 is used to store the cold data into the disk of the local memory area.

[0135] In one possible implementation, the type of non-shared data includes hot data, which is used to indicate data that the network device accesses frequently.

[0136] The confirmation unit 303 is used to confirm that the non-shared data is the hot data;

[0137] The storage unit 302 is used to store the hot data in the cache of the local memory region.

[0138] In one possible implementation, the type of the target data includes shared data, which is used to indicate data that needs to be accessed by other network devices.

[0139] Confirmation unit 303 is used to confirm that the target data is the shared data;

[0140] The storage unit 302 is used to store the shared data in the shared memory area according to the type of the shared data.

[0141] In one possible implementation, the type of shared data includes persistent data.

[0142] The confirmation unit 303 is used to confirm that the shared data is the persistent data;

[0143] The storage unit 302 is used to store the persistent data in the non-volatile medium of the shared memory region.

[0144] In one possible implementation, the type of shared data includes non-persistent data.

[0145] The confirmation unit 303 is used to confirm that the shared data is the non-persistent data;

[0146] The storage unit 302 is used to store the non-persistent data in the volatile medium of the shared memory region.

[0147] In one possible implementation, the target memory pool includes a northbound interface, which includes a first-level interface, a second-level interface, and a third-level interface. The first-level interface includes a memory semantic interface, the second-level interface includes a distributed data structure interface, and the third-level interface includes an application semantic interface, a file semantic interface, and / or a programming model interface.

[0148] In one possible implementation, the memory semantic interface includes Big Memory, persistent multi-copy plog, and / or memory object Memobj.

[0149] In one possible implementation, the distributed data structure interface includes key-value pairs, hashes, B+ trees, arrays, and / or matrices.

[0150] One possible implementation is characterized by,

[0151] The application semantic interface includes Shuffle, Cache, WAL, Message Queue MQ and / or Parameter Servers (AIParameter Servers); the file semantic interface includes Posix, HDFS, MPI-IO and / or JavaFile; the programming semantic interface includes UPC, UPC++, OpenSHMEM and / or X10.

[0152] It should be noted that the information interaction and execution process between the modules of the network device 30 mentioned above are based on the same concept as the method example of this application, and the execution steps are consistent with the detailed content of the method steps mentioned above. Please refer to the description in the method example mentioned above.

[0153] The above examples provide a different implementation of network device 30. Below is a network device 40, such as... Figure 8 As shown, the network device 40 is used to perform the steps executed by the network device (computing node) in the above example. For details on the execution steps and the corresponding beneficial effects, please refer to the corresponding examples above. They will not be repeated here.

[0154] See Figure 8The diagram shown illustrates the structure of a network device 40 provided in this application. The network device 400 includes a processor 402, a communication interface 403, and a memory 401. Optionally, it may include a bus 404. The communication interface 403, processor 402, and memory 401 can be interconnected via the bus 404. The bus 404 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 8 The symbol is represented by only one thick line, but this does not indicate that there is only one bus or one type of bus. This network device 400 can achieve... Figure 7 The network device 30 in the example shown has the following functions. The processor 402 and the communication interface 403 can perform the corresponding operations of the network device in the above method example.

[0155] The following is combined with Figure 8 A detailed introduction to each component of a network device:

[0156] The memory 401 may be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, used to store program code, configuration files, or other content that can implement the method of this application.

[0157] The processor 402 is the control center of the controller. It can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the examples provided in this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0158] Communication interface 403 is used for communication with other devices.

[0159] The processor 402 can perform the aforementioned... Figure 7 The operations performed by network device 30 in the example shown are not detailed here.

[0160] It should be noted that the information interaction and execution process between the modules of the network device 400 mentioned above are based on the same concept as the method example of this application, and the execution steps are consistent with the detailed content of the method steps mentioned above. Please refer to the description in the method example mentioned above.

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data storage method, characterized in that, include: A network device acquires target data, and a target application runs on the network device. The target data is used to indicate the data that the target application needs to store. The network device stores the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory area and a shared memory area. The local memory area is used to indicate the local memory resources of the network device, and the shared memory area is used to indicate the logical memory composed of the memory resources of the network device and / or multiple other network devices. When the type of the target data includes non-shared data, the non-shared data is used to indicate data that does not require access by other network devices. The network device stores the target data in the target memory pool according to the type of the target data, including: The network device confirms that the target data is the non-shared data; The network device stores the non-shared data in the local memory area according to the type of the non-shared data; When the type of the target data includes shared data, the shared data is used to indicate data that needs to be accessed by other network devices. The network device stores the target data in a target memory pool according to the type of the target data, including: The network device confirms that the target data is the shared data; The network device stores the shared data in the shared memory area according to the type of the shared data.

2. The data storage method according to claim 1, characterized in that, The types of non-shared data include cold data, which indicates data that is accessed infrequently by the network device. The network device stores the non-shared data in its local memory area according to its type, including: The network device confirms that the non-shared data is the cold data; The network device stores the cold data on the disk in the local memory area.

3. The data storage method according to claim 1, characterized in that, The types of non-shared data include hot data, which indicates data accessed frequently by the network device. The network device stores the non-shared data in its local memory area according to the type of non-shared data, including: The network device confirms that the non-shared data is the hot data; The network device stores the hot data in the cache of the local memory region.

4. The data storage method according to claim 1, characterized in that, The shared data includes persistent data, and the network device stores the shared data in the shared memory area according to the type of the shared data, including: The network device confirms that the shared data is the persistent data; The network device stores the persistent data in the non-volatile medium of the shared memory region.

5. The data storage method according to claim 1, characterized in that, The shared data includes non-persistent data, and the network device stores the shared data in the shared memory area according to the type of the shared data, including: The network device confirms that the shared data is non-persistent data; The network device stores the non-persistent data in the volatile medium of the shared memory region.

6. The data storage method according to any one of claims 1 to 5, characterized in that, The target memory pool includes a northbound interface, which includes a first-level interface, a second-level interface, and a third-level interface. The first-level interface includes a memory semantic interface, the second-level interface includes a distributed data structure interface, and the third-level interface includes an application semantic interface, a file semantic interface, and / or a programming model interface.

7. The network device according to claim 6, characterized in that, The target application schedules the third-level interface by selecting a specific interface object from the third-level interface. The third-level interface schedules the second-level interface by selecting a specific interface object from the second-level interface. The second-level interface schedules the first-level interface by selecting a specific interface object from the first-level interface. The first-level interface is used to schedule the target memory pool.

8. The data storage method according to claim 7, characterized in that, The memory semantic interface includes Big Memory, persistent multi-copy plog, and / or memory object Memobj.

9. The data storage method according to claim 6, characterized in that, The distributed data structure interface includes key-value pairs, hash tables, B+ trees, arrays, and / or matrices.

10. The data storage method according to claim 6, characterized in that, The application semantic interfaces include Shuffle, Cache, WAL, Message Queue (MQ), and / or Parameter Servers (AIParameter Servers). The file semantic interface includes Posix, HDFS, MPI-IO, and / or Java File; The programming model interface includes UPC, UPC++, OpenSHMEM, and / or X10.

11. A network device, characterized in that, include: An acquisition unit is used to acquire target data, wherein a target application is running on the network device, and the target data is used to indicate the data that the target application needs to store. A storage unit is used to store the target data in a target memory pool according to the type of the target data. The target memory pool includes a local memory area and a shared memory area. The local memory area is used to indicate the local memory resources of the network device, and the shared memory area is used to indicate the logical memory composed of the memory resources of the network device and / or multiple other network devices. When the type of the target data includes non-shared data, the non-shared data is used to indicate data that does not require access from other network devices. A confirmation unit is used to confirm that the target data is the non-shared data; The storage unit is used to store the non-shared data in the local memory area according to the type of the non-shared data; When the type of the target data includes shared data, the shared data is used to indicate data that needs to be accessed by other network devices. The confirmation unit is used to confirm that the target data is the shared data; The storage unit is used to store the shared data in the shared memory area according to the type of the shared data.

12. The network device according to claim 11, characterized in that, The types of non-shared data include cold data, which is used to indicate data that the network device accesses infrequently. The confirmation unit is used to confirm that the non-shared data is the cold data; The storage unit is used to store the cold data to the disk in the local memory area.

13. The network device according to claim 11, characterized in that, The types of non-shared data include hot data, which is used to indicate data that the network device accesses frequently. The confirmation unit is used to confirm that the non-shared data is the hot data; The storage unit is used to store the hot data in the cache of the local memory region.

14. The network device according to claim 11, characterized in that, The types of shared data include persistent data. The confirmation unit is used to confirm that the shared data is the persistent data; The storage unit is used to store the persistent data in the non-volatile medium of the shared memory region.

15. The network device according to claim 11, characterized in that, The types of shared data include non-persistent data. The confirmation unit is used to confirm that the shared data is the non-persistent data; The storage unit is used to store the non-persistent data in the volatile medium of the shared memory region.

16. The network device according to any one of claims 11 to 15, characterized in that, The target memory pool includes a northbound interface, which includes a first-level interface, a second-level interface, and a third-level interface. The first-level interface includes a memory semantic interface, the second-level interface includes a distributed data structure interface, and the third-level interface includes an application semantic interface, a file semantic interface, and / or a programming model interface.

17. The network device according to claim 16, characterized in that, The target application schedules the third-level interface by selecting a specific interface object from the third-level interface. The third-level interface schedules the second-level interface by selecting a specific interface object from the second-level interface. The second-level interface schedules the first-level interface by selecting a specific interface object from the first-level interface. The first-level interface is used to schedule the target memory pool.

18. The network device according to claim 17, characterized in that, The memory semantic interface includes BigMemory, persistent multi-copy plog, and / or memory object Memobj.

19. The network device according to claim 17 or 18, characterized in that, The distributed data structure interface includes key-value pairs, hash tables, B+ trees, arrays, and / or matrices.

20. The network device according to claim 16, characterized in that, The application semantic interfaces include Shuffle, Cache, Write-Ahead Log (WAL) system, Message Queue (MQ) and / or Parameter Servers (AI). The file semantic interface includes the portable operating system interface POSIX, the distributed file system HDFS, the information transfer interface MPI-IO, and / or Java File; The programming model interface includes UPC, UPC++, OpenSHMEM, and / or X10.

21. A network device, characterized in that, include: Processor, memory, and communication interface; The processor is connected to the memory and the communication interface; The communication interface is used to communicate with other devices; The processor reads instructions stored in the memory and causes the network device to execute the method as described in any one of claims 1 to 10.

22. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 10.

23. A communication system, said communication system comprising at least a network device; The network device is the network device included in the data storage method according to any one of claims 1 to 10.

24. A converged memory system, the converged memory system comprising multiple network devices, the network devices using a target memory pool to store target data; The target memory pool includes a local memory region and a shared memory region. The local memory region includes a disk and a cache, and the shared memory region includes volatile media and non-volatile media. When the type of the target data includes non-shared data, the non-shared data is used to indicate data that does not require access by other network devices, and the local memory area is used to store the non-shared data determined by the network device according to the type of the non-shared data; when the type of the target data includes shared data, the shared data is used to indicate data that requires access by other network devices, and the shared memory area is used to store the shared data determined by the network device according to the type of the shared data.

25. A converged memory system, the converged memory system comprising a first network device and a plurality of second network devices, wherein the first network device uses a target memory pool to store target data; The target memory pool includes a local memory region and a shared memory region. The local memory region is used to indicate the local memory of the first network device and includes disks and caches. The shared memory region is used to indicate the logical memory constructed after pooling memory resources in multiple second network devices and includes volatile media and non-volatile media. When the type of the target data includes non-shared data, the non-shared data is used to indicate data that does not require access by other network devices, and the local memory area is used to store the non-shared data determined by the first network device according to the type of the non-shared data; when the type of the target data includes shared data, the shared data is used to indicate data that requires access by other network devices, and the shared memory area is used to store the shared data determined by the first network device according to the type of the shared data.

Images