Data processing method and related device

Abstract

The invention discloses a data processing method and a related device, and the data processing method comprises the steps: firstly, receiving a writing request for generating target data by computing equipment; then, the computing device writes the target data into a first memory bound with the sending network card and obtains address information of the target data in a second memory, the sending network card sends the target data to the second memory of the storage device according to a logic sequence, and the sending network card sends the target data in a remote memory access mode; the second memory is a power-preserving memory; and finally, the computing device sends address information to the storage device, and the address information is used for the storage device to store the target data from the second memory into the memory according to the logic sequence. Therefore, the copying times of the data are reduced, the occupation of processing resources is reduced, and the utilization rate of system resources is improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method and related devices. Background Art

[0002] In the scenario where computing devices and storage devices are deployed separately, since the input / output (I / O) rate of the computing device does not match the storage device's disk download rate, if the computing device directly sends the data to the storage device and downloads it to the disk after receiving a data write request, the data may not be downloaded in time. In order to solve the above problems, after receiving a data write request, the computing device generally sends the data to the memory of the storage device, and the storage device temporarily stores the data in the memory in a non-power-off volatile storage medium such as Write-ahead logging (WAL), and then downloads it to the disk later.

[0003] However, when the storage device is downloading data to disk, it needs to move the data from the memory to the WAL medium, and then move the data from the WAL medium to the final storage, resulting in a large number of data copies and occupying more processing resources. Summary of the invention

[0004] The present application provides a data processing method and related devices, which can reduce the number of data copies and resource occupation.

[0005] In a first aspect, the present application provides a data processing method, which can be applied to the field of storage technology, and the method includes:

[0006] First, the computing device receives a write request for generating target data; then, the computing device writes the target data into a first memory bound to a sending network card, and obtains address information of the target data in a second memory, wherein the sending network card sends the target data to the second memory of the storage device in a logical order, the sending network card uses a remote memory access method to send the target data, and the second memory is a power-saving memory; finally, the computing device sends address information to the storage device, and the address information is used by the storage device to store the target data from the second memory to the storage device in a logical order.

[0007] In the present application, on the first aspect, after generating a write request for the target data, the computing device temporarily stores the target data in the second memory that maintains power, thereby ensuring that the data in the memory will not be lost, and also solving the problem of the mismatch between the input / output rate of the computing device and the off-disk rate of the storage device. On the second aspect, the computing device and the storage device adopt remote memory access technology, and communicate data between the network card and the second memory of the storage device without notifying the CPU of the storage device for processing, thereby reducing the occupancy of the processor of the storage device and the performance consumption of the CPU, and also reducing the communication delay between the computing device and the storage device, thereby improving the efficiency of data writing. On the third aspect, the computing device writes the target data to the first memory in a logical order, and the storage device reads the data from the second memory in a logical order, thereby ensuring the logical order between the memory shards. On this basis, the storage device can directly read the data from the second memory and off-disk in an orderly manner, thereby reducing the number of data copies, reducing the occupancy of processing resources, and improving the utilization of system resources. Fourthly, the computing device binds the first memory storing the target data to the sending network card, so that the computing device can transfer the target data in the first memory to the second memory of the storage device based on the sending network card, thereby avoiding the problem of data being destroyed due to other processes accessing the first memory.

[0008] In a possible implementation manner of the first aspect, the sending network card sends the target data to the second memory of the storage device in a logical order, including:

[0009] The sending network card receives the first information sent by the receiving network card, where the first information includes address information of the second memory arranged in a logical order;

[0010] The sending network card sends the target data to the second memory of the receiving network card according to the first information.

[0011] In this implementation, the sending network card moves the target data according to the logical order of the second memory, thereby ensuring the orderly writing of the data.

[0012] In a possible implementation manner of the first aspect, the method further includes:

[0013] The computing device generates a read request for target data;

[0014] When the target data is in the second memory, the computing device obtains the target data from the second memory.

[0015] At present, when it is determined that the target data is not yet stored in the power-saving memory, the data is generally taken out from the WAL, copied to the memory of the storage device, and then sent to the computing device, resulting in a large number of data copies and a large number of resources occupied. In this implementation, after the computing device generates a read request for the target data, if it is determined that the target data is in the second memory and has not yet been stored, the data can be directly read from the second memory, thereby reducing the number of data copies and resource occupation, and improving the utilization of system resources.

[0016] In a possible implementation manner of the first aspect, the sending network card is a remote direct memory access (RDMA) network card, and the sending network card uses a unilateral RDMA protocol to send the target data to the second memory of the storage device.

[0017] In this implementation, compared to using a network protocol for communication, the computing device and the storage device use an RDMA network card, and realize data communication with the second memory of the storage device through the sending network card, without notifying the CPU of the storage device for processing, thereby reducing the occupancy of the processor of the storage device and the performance consumption of the CPU, and also reducing the communication delay between the computing device and the storage device, and improving the efficiency of data writing. In addition, the computing device binds the first memory storing the target data to the sending network card, which enables the computing device to transfer the target data in the first memory to the second memory of the storage device based on the sending network card, avoiding the problem of data being destroyed due to other processes accessing the first memory.

[0018] In a possible implementation manner of the first aspect, the sending network card uses a Compute Express Link (CXL) protocol to send the target data to the second memory of the storage device.

[0019] In this implementation, the sending network card uses the CXL protocol to send the target data to the second memory of the storage device, which can reduce the occupancy and performance consumption of the processor of the storage device.

[0020] In a second aspect, the present application provides a data processing method, characterized by comprising:

[0021] First, the storage device receives address information of target data sent by the computing device, wherein the address information is used to indicate the location of the target data in the second memory, and the target data is stored in the second memory of the receiving network card in a logical order. The receiving network card receives the target data by remote memory access, and the second memory is a power-saving memory; then, the storage device stores the target data from the second memory to the storage device according to the address information.

[0022] In the present application, on the first hand, the storage device can obtain the target data from the second memory of the receiving network card according to the address information of the target data sent by the computing device, without notifying the CPU of the storage device to process, thereby reducing the occupancy of the processor of the storage device and the performance consumption of the CPU. On the second hand, since the second memory is a power-saving memory, after the storage device obtains the target data from the second memory, it can directly store the data in the memory, thereby reducing the number of data copies and improving the data writing efficiency. On the third hand, after receiving the target data, the receiving network card stores the target data in the second memory in a logical order, thereby ensuring the orderly writing of the data.

[0023] In a possible implementation manner of the second aspect, the storage device stores the target data from the second memory to the storage according to the address information, including:

[0024] The storage device obtains the target data according to the address information;

[0025] The storage device performs a deserialization operation on the target data to obtain the deserialized target data;

[0026] The storage device writes the deserialized target data into the memory.

[0027] In this implementation, after the computing device serializes the target data, since the target data may contain data corresponding to multiple requests, after the computing device serializes the multiple requests contained in the target data in sequence and sends them to the storage side, the storage device needs to deserialize each request in sequence before storing it in the memory. In addition, a write request may contain multiple fields, and the parsing of different fields of a request requires deserialization.

[0028] In a possible implementation manner of the second aspect, the storage device stores the target data from the second memory to the storage according to the address information, including:

[0029] The storage device obtains the target data according to the address information;

[0030] The storage device sends data exchange instructions to the second memory and the storage device respectively, so that the data in the second memory and the storage device are exchanged.

[0031] In this implementation, the storage device sends a data exchange instruction to the second memory and the disk, so that the data in the second memory and the disk are exchanged without notifying the CPU to process the data off the disk, thereby reducing CPU consumption.

[0032] In a possible implementation manner of the second aspect, the storage device acquires target data according to the address information, including:

[0033] The storage device determines the location of the memory slice where the target data is located in the second memory according to the address information;

[0034] The storage device determines the readable range based on the location of the memory slice;

[0035] The storage device obtains the target data from the readable range in a logical order.

[0036] In this implementation, the storage device can determine the readable range according to the address information and read the target data in a logical order, thereby ensuring that the data is stored in an orderly manner.

[0037] In a possible implementation manner of the second aspect, the receiving network card is a unilateral remote direct memory access (RDMA) network card, and the receiving network card uses the RDMA protocol to receive target data sent by the computing device.

[0038] In a possible implementation manner of the second aspect, the receiving network card uses a computing fast link CXL protocol to receive target data sent by the computing device.

[0039] In a third aspect, the present application provides a computing device, and the beneficial effects can be found in the description of the first aspect, which will not be repeated here. The computing device has the function of implementing the behavior in the method example of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the computing device includes:

[0040] A generating module, used for generating a write request for target data;

[0041] The processing module is used to write the target data into the first memory bound to the sending network card, and obtain the address information of the target data in the second memory, the sending network card sends the target data to the second memory of the storage device in a logical order, the sending network card sends the target data in a remote memory access mode, and the second memory is a power-saving memory;

[0042] The transceiver module is used to send address information to the storage device, and the address information is used by the storage device to store the target data from the second memory to the storage device in a logical order.

[0043] The generation module, processing module and transceiver module can perform the corresponding functions in the above-mentioned first aspect method example. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0044] In a possible implementation manner of the third aspect, the processing module is further configured to:

[0045] The sending network card receives the first information sent by the receiving network card, where the first information includes address information of the second memory arranged in a logical order;

[0046] The sending network card sends the target data to the second memory of the receiving network card according to the first information.

[0047] In a possible implementation of the third aspect, the generating module is further configured to generate a read request for the target data;

[0048] The processing module is also used for acquiring the target data from the second memory when the target data is in the second memory.

[0049] In a possible implementation manner of the third aspect, the sending network card is a remote direct memory access (RDMA) network card, and the sending network card uses a unilateral RDMA protocol to send the target data to the second memory of the storage device.

[0050] In a possible implementation manner of the third aspect, the sending network card uses a Compute Express Link (CXL) protocol to send the target data to the second memory of the storage device.

[0051] In a fourth aspect, the present application provides a storage device, and the beneficial effects can be found in the description of the second aspect, which will not be repeated here. The storage device has the function of implementing the behavior in the method example of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the storage device includes:

[0052] A transceiver module, used for receiving address information of target data sent by a computing device, the address information is used for indicating the location of the target data in a second memory, the target data is stored in the second memory of a receiving network card in a logical order, the receiving network card receives the target data in a remote memory access manner, and the second memory is a power-saving memory;

[0053] The processing module is used to store the target data from the second memory into the storage device according to the address information.

[0054] The transceiver module and the processing module can perform the corresponding functions in the above-mentioned second aspect method example. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0055] In a possible implementation manner of the fourth aspect, the processing module is further configured to:

[0056] The storage device obtains the target data according to the address information;

[0057] The storage device performs a deserialization operation on the target data to obtain the deserialized target data;

[0058] The storage device writes the deserialized target data into the memory.

[0059] In a possible implementation manner of the fourth aspect, the processing module is further configured to:

[0060] The storage device obtains the target data according to the address information;

[0061] The storage device sends data exchange instructions to the second memory and the storage device respectively, so that the data in the second memory and the storage device are exchanged.

[0062] In a possible implementation manner of the fourth aspect, the processing module is further configured to:

[0063] The storage device determines the location of the memory slice where the target data is located in the second memory according to the address information;

[0064] The storage device determines the readable range based on the location of the memory slice;

[0065] The storage device obtains the target data from the readable range in a logical order.

[0066] In a possible implementation manner of the fourth aspect, the receiving network card is a unilateral remote direct memory access (RDMA) network card, and the receiving network card uses the RDMA protocol to receive target data sent by the computing device.

[0067] In a possible implementation manner of the fourth aspect, the receiving network card uses the Compute Express Link (CXL) protocol to receive the target data sent by the computing device.

[0068] In a fifth aspect, the present application provides a computing device, the computing device comprising a processor, the processor running program instructions in a memory to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0069] In a sixth aspect, the present application provides a storage device, including a processor, wherein the processor runs program instructions in a memory to execute the method in the above-mentioned second aspect or any possible implementation manner of the second aspect.

[0070] In a seventh aspect, the present application provides a storage system including a computing device and a storage device.

[0071] The computing device is used to execute the method in the first aspect or any possible implementation manner of the first aspect;

[0072] The storage device is used to execute the method in the above-mentioned second aspect or any possible implementation manner of the second aspect.

[0073] In an eighth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the method in any possible implementation of the first to second aspects above.

[0074] In a ninth aspect, the present application provides a chip system, which includes a processor for implementing the method in any possible implementation of the first aspect to the second aspect. In a possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0075] The solutions of the second to ninth aspects mentioned above are used to implement or cooperate with the method of the first aspect or any possible implementation method thereof, and therefore can achieve the same or corresponding beneficial effects as the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of the structure of a storage system provided in an embodiment of the present application;

[0077] Figure 2 Another structural schematic diagram of a storage system provided in an embodiment of the present application;

[0078] Figure 3a A schematic diagram of the architecture of the currently used data processing method;

[0079] Figure 3b Another schematic diagram of the current data processing method

[0080] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0081] Figure 5 Another flowchart of the data processing method provided in the embodiment of the present application;

[0082] Figure 6a A schematic diagram of an architecture for currently used computing devices and storage devices to communicate;

[0083] Figure 6b A schematic diagram of an architecture for communication between a computing device and a storage device provided in an embodiment of the present application;

[0084] Figure 7a A schematic diagram of an architecture for applying the data processing method provided in an embodiment of the present application;

[0085] Figure 7b Another schematic diagram of the architecture of the data processing method provided in the embodiment of the present application;

[0086] Figure 8 Another flowchart of the data processing method provided in the embodiment of the present application;

[0087] Fig. 9A schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0088] Fig.10 A schematic diagram of a structure of a storage device provided in an embodiment of the present application;

[0089] Fig.11 Another structural schematic diagram of a computing device provided in an embodiment of the present application;

[0090] Fig.12 Another structural schematic diagram of a storage device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0092] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0093] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0094] It should also be noted that, in some alternative implementations, the functions / acts noted may occur out of the order of the drawings. For example, two figures shown in succession may in fact occur substantially simultaneously or may sometimes be performed in the reverse order, depending on the functions / acts involved.

[0095] In the embodiments of the present application, unless otherwise specified, the meaning of "at least one" refers to one or more, and the meaning of "plurality" refers to two or more. It is understood that in the present application, "when", "if" and "if" all refer to the device making corresponding processing under certain objective circumstances, and do not limit the time, nor do they require that there must be a judgment action when the device is implemented, nor do they mean that there are other limitations. In addition, the special word "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0096] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0097] First of all, in order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts that may be involved in the embodiments of the present application are introduced below.

[0098] 1. Remote Direct Memory Access

[0099] Remote direct memory access (RDMA) is a technology that bypasses the operating system kernel of a remote device (such as a storage device) to access data in its memory. Since it does not go through the operating system, it not only saves a lot of CPU resources, but also improves system throughput and reduces the system's network communication latency. It is particularly suitable for widespread application in large-scale parallel computer clusters.

[0100] RDMA has several major characteristics: (1) Data is transmitted between remote devices over the network; (2) There is no involvement of the operating system kernel, and all content related to sending and transmitting is offloaded to the smart network card; (3) Data is directly transmitted between the user space virtual memory and the smart network card without involving the operating system kernel, and there is no additional data movement and copying.

[0101] 2. Write the pre-log

[0102] In the computer field, write-ahead logging (WAL) is a way of recording information to ensure the atomicity and persistence of data processing operations. The write-ahead log can record all modification operations on the data. For example, the write-ahead log can include redo logs and undo logs.

[0103] When data needs to be written to a storage device, one current operation method is to write the data to the storage device, such as writing it to the WAL of the storage device. Among them, WAL can be understood as a space that can only append data and consume data in the past, and the carrier of this space can be a power-saving memory or a file, etc. The writing method of WAL can adopt the append writing method, that is, starting to write the next write-before log that needs to be written at the end of the last write-before log. It can be understood that the writing method of the write-before log is a sequential writing method, and the writing (or generation) order of each write-before log can be determined according to the writing position of each write-before log that has been written.

[0104] In order to better understand the solution of the embodiment of the present application, the application scenario of the embodiment of the present application is described below. The data processing method provided in the embodiment of the present application is mainly applied to the computing device and storage device of the storage system. The computing device and the storage device can be connected in a one-to-one, one-to-many, many-to-one or many-to-many manner.

[0105] This embodiment is described by taking the storage system as a distributed storage system as an example. Figure 1 , Figure 1 A schematic diagram of a storage system provided in an embodiment of the present application. Figure 1 As shown, the distributed storage system 100 includes a computing device cluster and a storage device cluster.

[0106] Users access data through applications. Computers running these applications can be called "client devices". Client devices can be physical machines or virtual machines. Physical client devices include but are not limited to desktop computers, servers, laptops, and mobile devices. Users can initiate data access requests, such as data write requests or data read requests, to the computing device 110 in the storage system 100 through the client device. The functions of the client device can also be integrated into the computing device.

[0107] The computing device cluster includes one or more computing devices 110, Figure 1 Two computing devices 110 are shown in the figure, but are not limited to two computing devices 110. Each computing device 110 can communicate with each other through a network. The computing device 110 is a computing device that is a node and hub of network data and is responsible for providing information services to multiple clients or users in the network at the same time, such as a server, a computer, a storage array controller, etc.

[0108] The computing device 110 includes at least a processor 112 , a memory 113 , and a network card 114 .

[0109] Processor 112 is a general-purpose central processing unit (CPU) for processing data access requests from outside computing device 110 or requests generated inside computing device 110. Processor 112 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices (PLD), transistor logic devices, hardware components or any combination thereof. The above-mentioned PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0110] Figure 1 Only one processor 112 is shown. In actual applications, there may be multiple processors 112 in a computing device 110, wherein a processor 112 may have one or more processor cores. This embodiment does not limit the number of processors and processor cores.

[0111] Memory 113 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very fast speed. It serves as a temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be random access memory (RAM), dynamic random access memory (DRAM), storage class memory (SCM), static random access memory (SRAM), etc. It can also be read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), etc.

[0112] In addition, the memory 113 may also be a dual inline memory module or a dual inline memory module (DIMM), that is, a module composed of DRAM, or a solid state disk (SSD).

[0113] In actual applications, the computing device 110 may be configured with multiple memories 113 and memories 113 of different types. This embodiment does not limit the number and type of the memories 113. In addition, the memory 113 may be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 113 will not be lost. The memory with the power-saving function is called a non-volatile memory.

[0114] The network card 114 is used to communicate with the storage device 120. For example, when the total amount of data in the memory 113 reaches a certain threshold, the computing device 110 may send a request to the storage device 120 through the network card 114 to perform persistent storage on the data.

[0115] In addition, the computing device 110 may also include a bus for communication between components within the computing device 110. Figure 1 The main function of the computing device 110 in the example is computing services. When storing data, it can use remote storage to achieve persistent storage. Therefore, it has less local storage than a conventional server, thereby saving costs and space. However, this does not mean that the computing device 110 cannot have local storage. In actual implementation, the computing device 110 can also have a small number of hard disks built in or externally connected. Any computing device 110 can access any storage device 120 in the storage device cluster through the network.

[0116] The storage device cluster may include multiple storage devices 120, which are used to persistently store data transmitted by the computing device. The storage device may be a distributed storage system, or a distributed storage device or a distributed storage server in a distributed storage system. Figure 1 Two storage devices 120 are shown in FIG. 1 , but the present invention is not limited to two storage devices 120. The storage device may also be a centralized storage device or a storage server.

[0117] A storage device 120 includes one or more control units 121, a network card 124 and multiple hard disks 125. The network card 124 is used to communicate with the computing device 110. The hard disk 125 is used to store data and can be a disk or other types of storage media, such as solid state drives (SSD, also known as solid-state devices, solid-state disks or flash drives), mechanical hard disks (HDD), magnetic tapes, optical disks, etc. The control unit 121 is used to write data to the hard disk 125 or read data from the hard disk 125 according to the read / write data request sent by the computing device 110. In the process of reading / writing data, the control unit 121 needs to convert the address carried in the read / write data request into an address that the hard disk can recognize. It can be seen that the control unit 121 also has some simple computing functions.

[0118] In practical applications, the control unit 121 may have various forms. In one case, the control unit 121 includes a processor and a memory. The processor is a general-purpose central processing unit (CPU) for processing data access requests from outside the computing device 110 or requests generated inside the computing device 110. The processor 112 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices (PLD), transistor logic devices, hardware components or any combination thereof. The above-mentioned PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The processor is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 125, or data read from the hard disk 125 to be sent to the computing device 110.

[0119] Optionally, the number of control units 121 may be one, or may be two or more.

[0120] Optional, Figure 1The structure of the storage device shown is a disk-controller integrated architecture, that is, the control unit 121 and the hard disk 125 are included in the same physical device, but in some embodiments, the structure of the storage device can also be a disk-controller separated structure. This embodiment is mainly described by taking the storage device as a disk-controller integrated architecture as an example.

[0121] In some possible embodiments, the storage device 120 may be provided to the user as a cloud storage service. Specifically, the storage device 120 may run in a cloud environment, for example, on a public cloud, a private cloud, or a hybrid cloud. The user may use a terminal to apply for a certain capacity of storage space in the cloud storage service. The computing device 110 may allocate storage space of corresponding capacity to the user, for example, allocating one or more storage devices 120 to the user, so as to store data in the allocated storage space. As an example, the storage device 120 may be provided as an object storage service (OBS), a cloud hard disk, a cloud database, etc.

[0122] Understandably, Figure 1 The diagram shows the architecture of a distributed storage system. The solution of the embodiment of the present application can also be applied to a centralized storage system. In a centralized storage system, data of a computing device is stored in a storage device, which can be, for example, a storage array.

[0123] See also Figure 2 , Figure 2 Another structural schematic diagram of a storage system provided in an embodiment of the present application. Figure 2 The storage system shown is a centralized storage system 200. The storage device 120 and the computing device 110 have the same structure as Figure 1 The same as in , I will not repeat them here.

[0124] The data processing method provided in the embodiment of the present application is mainly suitable for scenarios where computing storage or computing storage is deployed separately. It can be applied to separate storage systems and centralized storage systems, and can also be applied to database systems.

[0125] The embodiments of the present application do not limit the specific form of the computing device for data processing. The computing device is, for example, a mobile phone terminal, a tablet computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, or a vehicle terminal. The computing device can also be an edge device (for example, a box carrying a chip with processing capabilities), a server, a device with computing capabilities in a distributed storage system, or a device with computing capabilities in a centralized storage system.

[0126] In the above application scenarios, in order to solve the problem of mismatch between the input / output (I / O) rate of the computing device and the rate of the storage device to be written to the disk, a write ahead log (WAL) is generally used to store the front-end data in the WAL, and then replay the write ahead log on the back end, thereby improving the I / O write efficiency. The purpose of WAL is to balance the rate of front-end writing I / O and back-end data being written to the disk by storing the data in a medium space whose performance is non-volatile due to power failure of the hard disk. It can be understood that the main purpose of the write ahead log is to solve the problem of unequal rates of front-end computing device writing IO and back-end storage device writing data to the disk by storing the data in a non-volatile medium space with better performance than the memory.

[0127] WAL has the following two basic characteristics:

[0128] (1) The data written to WAL must be guaranteed not to be lost. Therefore, the space occupied by WAL must be a non-volatile medium;

[0129] (2) Data can only be written to WAL in an append-only manner to ensure data orderliness.

[0130] Based on the different communication methods between computing devices and storage devices, there are generally the following two data processing methods.

[0131] See also Figure 3a , Figure 3a This is a schematic diagram of the architecture of the data processing method currently used. Figure 3a As shown, the computing device and the storage device communicate using the transmission control protocol / internet protocol (TCP / IP). After the computing device sends the received data to the TCP / IP driver network card of the storage device through the TCP / IP driver network card, the storage device needs to move the data from the memory to the WAL medium, and then move the data from the WAL medium to the final disk, resulting in more data copies and more occupied processing resources.

[0132] In addition, a large amount of data occupies the communication bandwidth between the computing device and the storage device, resulting in high latency, and occupies the CPU on the host side and the storage side to process the communication protocol stack, reducing the device processing performance. There is network latency in the data transmission between the computing device and the storage device, and the order of data writing cannot be guaranteed. In order to ensure the order of writing WAL, the computing device needs to process communication transactions, which increases the difficulty of data processing.

[0133] See also Figure 3b , Figure 3b FIG. 1 is another schematic diagram of the architecture of the currently used data processing method. Figure 3b As shown in the figure, the computing device and the storage device communicate using the RDMA protocol. After the computing device sends the received data to the RDMA network card of the storage device through the RDMA network card, the storage device needs to move the data from the memory to the WAL medium, and then move the data from the WAL medium to the final disk, resulting in more data copies and more occupied processing resources.

[0134] from Figure 3a and Figure 3b It can be seen that when the computing device and the storage device communicate using the TCP / IP protocol, the CPUs of both the computing device and the storage device participate in the processing of the protocol stack in the memory. In addition, whether using TCP / IP technology or RDMA technology, the CPU of the storage device needs to move data from the memory to the WAL medium, and the CPU of the storage device needs to move data from the WAL medium to the final disk medium.

[0135] In order to solve the above problems, the present application provides a data processing method, which can be applied to Figure 1 The storage system shown can also be applied to systems such as databases, which are mainly executed by computing devices. For details, please refer to Figure 4 , Figure 4 A flow chart of a data processing method provided for an embodiment of the present application. The method specifically includes: S1. A computing device generates a write request for target data. S2. The computing device writes the target data into the first memory of a sending network card, and obtains the address information of the target data in the second memory. The sending network card sends the target data to the second memory of the storage device in a logical order. The sending network card sends the target data in a remote memory access manner, and the second memory is a power-saving memory. S3. The computing device sends the address information to the storage device, and the address information is used by the storage device to store the target data from the second memory to the storage device in the logical order.

[0136] It is understandable that the embodiment of the present application combines the functions of the memory registered into RDMA and WAL into one to reduce the processing occupation of the CPU and improve the utilization of system resources. In order to realize the two basic features of the above-mentioned WAL, in the embodiment of the present application, the second memory registered into RDMA is set as a power-saving memory to ensure that the data written into the second memory will not be lost. At the same time, by logically sorting the memory slices registered into RDMA, the logical precedence between the memory slices is guaranteed to ensure the orderliness of the written data.

[0137] The above introduces the application scenarios of the data processing method provided in the embodiments of the present application. The following will introduce in detail the implementation process of the data processing method provided in the embodiments of the present application.

[0138] For ease of explanation, the target data writing process and the target data reading process are described separately.

[0139] 1. Data writing process

[0140] See also Figure 5 , Figure 5 Another flow chart of the data processing method provided in the embodiment of the present application is shown in FIG. Figure 1 The computing device and storage device shown execute steps 501 to 505. For ease of description, the memory is described as a disk.

[0141] Step 501: A computing device generates a write request for target data.

[0142] Step 502: The computing device writes the target data into the first memory of the sending network card, and obtains the address information of the target data in the second memory.

[0143] In this embodiment, after generating a write request for the target data, the computing device writes the target data to be stored into the first memory of the sending network card. After the computing device writes the target data into the first memory, the sending network card uses remote memory access to send the target data in the first memory to the second memory of the storage device, and feeds back the address information of the target data in the second memory to the computing device. The second memory is a power-saving memory to ensure that the target data is not lost when the second memory loses power.

[0144] It can be understood that the embodiment of the present application combines the functions of the second memory and WAL into one, sets the second memory as a power-saving memory and ensures the orderly writing and reading of data to reduce the number of data copies. Correspondingly, in an optional implementation, the sending network card of the computing device receives the first information sent by the receiving network card of the storage device, wherein the first information includes the address information of the second memory arranged in a logical order. After the computing device generates a write request for the target data and writes the target data into the first memory, the sending network card sends the target data to the second memory of the receiving network card according to the first information.

[0145] In this implementation, the receiving network card will periodically send the address information of the second memory arranged in a logical order to the sending network card, and the sending network card will write the target data into the second memory of the receiving network card according to the logical order to ensure the orderly writing of the target data. Among them, the sending network card and the receiving network card will continuously negotiate the logical order of the memory slices of the second memory, and what is negotiated between the sending network card and the receiving network card is the logical order relationship between the memory slices in the second memory of the receiving network card. The logical order between the sending network card and the receiving network card is dynamically negotiated by itself without occupying CPU resources. Memory sharding refers to aligning memory addresses according to a fixed granularity. Using memory sharding is one of the ways to improve data processing efficiency, and it is also easy to manage memory usage.

[0146] Optionally, the computing device can determine the logical order between memory slices once when the memory is initialized. During the communication between the computing device and the storage device, the computing device and the storage device write and read data respectively according to the logical order, without the need to regularly negotiate and update the content of the logical order.

[0147] For example, see Figure 6a , Figure 6a The figure is a schematic diagram of an architecture for the communication between a computing device and a storage device currently used. In the current technology, after the computing device generates write request 1 and write request 2 in succession, it can send request 1 and request 2 to the second memory of the receiving network card through the sending network card. However, since there is no logical order between request 1 and request 2, that is, there is no logical order between memory shard 3 and memory shard 1, after the sending network card writes the data of request 1 and request 2 into memory 3 and memory 1 of the second memory respectively, it cannot be guaranteed that the storage device will first store the data of request 1 in the disk when moving the data in the second memory to the disk.

[0148] In this regard, in the embodiment of the present application, the computing device and the storage device write and read data respectively based on the negotiated logical order to ensure the orderly writing and downloading of data. Figure 6b , Figure 6b A schematic diagram of an architecture for communication between a computing device and a storage device provided in an embodiment of the present application. Figure 6bAs shown, the sending network card and the receiving network card will first negotiate the logical order relationship between the memory slices of the second memory. Assuming that the second memory includes n memory slices, and the logical order of the memory slices is 1, 3, n, 2, 0, the receiving network card can write request 1 and request 2 into the second memory respectively according to the logical order after receiving the first information sent by the sending network card. For example, the sending network card first writes request 1 into slice 1. If slice 1 is not full, request 2 will continue to be written into slice 1. If it is full, request 2 will be written into slice 3, and so on. Subsequently, the storage device can first move the data in request 1 from memory slice 1 of the second memory to the disk in the logical order, and then move the data of request 2 from memory slice 1 or 2 to the disk.

[0149] In an optional implementation, the sending network card is an RDMA network card, and the sending network card uses a unilateral RDMA protocol to send the target data to the second memory of the storage device.

[0150] In this implementation, when the sending network card is an RDMA network card, after the computing device writes the target data into the first memory bound to the RDMA network card, the RDMA network card can use the RDMA protocol to send the target data to the second memory of the storage device, thereby eliminating the need to notify the CPU of the storage device to process. Correspondingly, the second memory of the storage device is bound to the receiving network card of the storage device, and the receiving network card is an RDMA network card.

[0151] It is understandable that the first memory is a memory bound to the RDMA network card. In this embodiment, the computing device writes the target data into the first memory, and uses the RDMA network card to send the data in the first memory to the storage device based on the RDMA mechanism, thereby reducing the processor occupancy of the storage device and the performance consumption of the CPU.

[0152] In an optional implementation, the sending network card uses a computing fast link protocol to send the target data to the second memory of the storage device.

[0153] In this implementation, the sending network card can use the Compute Express Link (CXL) protocol to send data to the second memory of the storage device. Correspondingly, the sending network card is a network card device that can communicate using the CXL protocol.

[0154] It is understandable that the RDMA technology and CXL technology mentioned in this embodiment may also be replaced by other remote access technologies, which may be specifically set according to actual needs and are not limited here.

[0155] In an optional implementation, after receiving a write request for target data, the computing device may perform a serialization operation on the received target data to obtain serialized target data; then, the computing device writes the serialized target data into a first memory; finally, the computing device sends the serialized target data to a second memory of a storage device based on a sending network card, and obtains address information of the serialized target data in the second memory of the storage device.

[0156] In this implementation, when the target data is large or the number of write requests received is large, the computing device can perform serialization operations on the target data to save time in writing the data into the first memory and improve the efficiency of data writing. In addition, the serialization operation can ensure the order of data writing without the need for client devices or users to intervene. Figure 6b For example, assuming that the data volume of request 1 and request 2 is large, the computing device can first serialize the data in request 1 and request 2, and then write the serialized data into the first memory in a logical order.

[0157] When the sending network card sends the target data to the second memory of the storage device, the sending network card can determine the specific location where the target data is written to the second memory. Therefore, after writing the target data to the second memory, the sending network card can directly obtain the specific address information of the target data in the second memory. Since the sending network card sends the target data based on the remote access technology, after the sending network card sends the target data to the second memory of the receiving network card, the receiving network card will not inform the storage device that the new data has been received. Therefore, the sending network card will further feed back the address information to the computing device so that the computing device can inform the storage device of the location in the second memory where the target data has been written.

[0158] It can be understood that when the computing device writes the serialized data into the first memory, the starting address where the computing device writes into the first memory is known. Therefore, the address information corresponding to the serialized target data in the first memory can be deduced based on the offset address of a certain serialized data in the entire data block and the starting address in the first memory.

[0159] Optionally, the computing device may generate a first index relationship based on address information corresponding to the target data in the write request and address information corresponding to the target data in the first memory.

[0160] It is understandable that the main function of the first index relationship is to facilitate the computing device to read data. When the computing device receives a request to read the target data, if the target data has not yet been saved to the disk, the computing device can directly find the address information of the target data in the first memory based on the first index relationship and read it out. If there is no index information, when the computing device needs to read a certain data, it needs to traverse the address information of the data from the disk of the storage device, and the reading efficiency is low.

[0161] Step 503: The computing device sends address information of the target data in the second memory to the storage device.

[0162] In this embodiment, after the computing device writes the target data into the first memory bound to the sending network card, the sending network card can use remote memory access technology to send data to the second memory bound to the receiving network card of the storage device, and feedback the address information of the target data written into the second memory to the computing device.

[0163] It is understandable that, since the sending network card uses remote memory access technology to send data, the storage device cannot perceive after the sending network card writes data to the second memory bound to the receiving network card of the storage device. In order to enable the storage device to process the written data, after the sending network card sends data to the receiving network card and feeds back the written address information, the computing device can send the address information to the storage device, so that the storage device can further process the data written to the second memory based on the address information.

[0164] It is worth noting that after the computing device obtains the address information of the target data in the second memory, it can directly send the address information to the storage device without being processed by the upper-level application or client device, thereby improving the processing performance of the device and the data writing efficiency.

[0165] Step 504: the storage device receives the address information of the target data sent by the computing device.

[0166] In this embodiment, after receiving the address information fed back by the sending network card, the computing device will inform the storage device of the address information of the target data in the second memory, so that the storage device obtains the target data from the second memory according to the address information.

[0167] Optionally, the receiving network card is an RDMA network card, and the receiving network card uses the RDMA protocol to receive target data sent by the computing device.

[0168] Optionally, the receiving network card uses the CXL protocol to receive target data sent by the computing device.

[0169] It is understandable that the RDMA technology and CXL technology mentioned in this embodiment may also be replaced by other remote access technologies, which may be specifically set according to actual needs and are not limited here.

[0170] Step 505: The storage device stores the target data from the second memory to the disk according to the address information.

[0171] In this embodiment, after receiving the address information sent by the computing device, the storage device can move the target data from the second memory to the disk according to the address information.

[0172] In an embodiment of the present application, the storage device first obtains target data from the second memory according to the address information.

[0173] In an optional implementation, the storage device can determine the location of the memory slice where the target data is located in the second memory based on the address information, and then the storage device determines the readable range based on the location of the memory slice. Finally, the storage device obtains the target data from the readable range in a logical order.

[0174] In this implementation, the address information indicates the location of the memory slice where the target data is located in the second memory. Based on the address information, the storage device can determine from a logical order that the data in the memory slice and the previous memory slice can be played back, that is, the range that the storage device can read is determined. Then, when the storage device needs to play back, it can move the read target data to the disk from the beginning of the readable range in a logical order.

[0175] Exemplarily, assuming that the second memory includes n memory slices, and the logical order of the memory slices is 1, 3, n, 2, 0, and assuming that the address information indicates that the target data is on memory slice 3, the storage device can know that the data in slice 3 and slice 1 can be played back at any time after receiving the address information. When the storage device needs to play back, the storage device will read the data in the second memory starting from slice 1 in the logical order, then read the data in slice 3, and then store the data in slice 1 and slice 3 in the disk in sequence.

[0176] It is understandable that, in actual operation, some data may be accumulated in the second memory. The storage device reads data from the first shard in a logical order and can also play back the accumulated data.

[0177] Furthermore, after the storage device obtains the target data from the second memory, it will move the target data to the disk. Correspondingly, there are the following two moving methods.

[0178] In the first optional implementation, the computing device performs a serialization operation on the target data, and sends the serialized target data to the second memory of the storage device based on the sending network card. Correspondingly, after the disk obtains the target data according to the address information, the storage device performs a deserialization operation on the target data, and writes the deserialized data to the disk.

[0179] In this implementation, when the target data is large in size or the number of write requests received is large, the computing device can improve the data writing efficiency by serializing the target data. Correspondingly, the storage device needs to deserialize the target data and write the deserialized target data to the disk. Optionally, after the storage device writes the data to the disk, the storage device notifies the computing device that the target data has been written to the disk, so that the computing device releases the memory occupied by the target data.

[0180] It is understandable that after the computing device serializes the target data, since the target data may contain data of multiple requests, after the computing device serializes the multiple requests contained in the target data in sequence and sends them to the storage side, the storage device needs to deserialize each request in sequence before storing it in the memory. In addition, a write request may contain multiple fields, such as request type, request content, and request disk configuration (such as whether the disk is written to HDD or SSD, etc.), and the different field parsing of a request requires deserialization to solve.

[0181] In the first implementation, in order to provide a more intuitive introduction to the execution process of this embodiment, please refer to Figure 7a , Figure 7a A schematic diagram of an architecture of the data processing method provided in the embodiment of the present application. For ease of explanation, it is assumed that Figure 5 Step 502 is shown to be performed by the first module, and step 505 is performed by the second module.

[0182] like Figure 7a As shown, the computing device generates a write request for the target data, and writes the target data into the first memory bound to the sending network card based on the first module, and obtains the address information of the target data in the second memory. Then, the computing device sends the address information of the target data in the second memory to the storage device based on the sending network card. The receiving network card of the storage device receives the target data and obtains the target data from the second memory according to the address information. Furthermore, the storage device performs a deserialization operation on the target data based on the second module, and writes the deserialized target data into the disk.

[0183] In a second optional implementation, after receiving the address information, the storage device sends data exchange instructions to the second memory and the disk respectively, so that the data in the second memory and the disk are exchanged, thereby completing the data download to the disk.

[0184] In this implementation, after receiving the address information, the storage device can send data exchange instructions to the second memory and the disk respectively, and the second memory and the disk will exchange data with the support of the drive device, and write the target data at the specified address information of the second memory into the free position of the disk. The data exchange process is not processed by the processor of the storage device.

[0185] Optionally, after completing the data exchange, the storage device updates the index of the target data in the disk.

[0186] Optionally, after completing the data exchange, the storage device notifies the computing device that the target data has been played back, and the computing device releases the memory occupied by the target data.

[0187] In the second implementation, in order to provide a more intuitive introduction to the execution process of this embodiment, please refer to Figure 7b , Figure 7b Another schematic diagram of the architecture of the data processing method provided in the embodiment of the present application. For the sake of convenience, it is assumed that Figure 5 Step 502 is shown to be performed by the first module, and step 505 is performed by the second module.

[0188] like Figure 7b As shown, the computing device generates a write request for the target data, and based on the first module, writes the target data into the first memory bound to the sending network card, and obtains the address information of the target data in the second memory. Then, the computing device sends the address information of the target data in the second memory to the storage device based on the sending network card. The receiving network card of the storage device receives the target data, and obtains the target data from the second memory according to the address information. Furthermore, the storage device sends data exchange instructions to the second memory and the disk respectively, so that the data in the second memory and the disk are exchanged, and the target data storage is migrated from the second memory to the disk. After the target data is downloaded to the disk, the second module updates the index of the target data in the disk.

[0189] It is understandable that in the first implementation, the storage device needs to extract the target data from the second memory, perform a deserialization operation, and then download it to the disk, which is equivalent to re-parsing the target data. In the second implementation, the target data in the second memory is directly migrated to the disk through the memory-to-disk swap technology, and this process is completed based on the drive hardware without CPU processing. The CPU only needs to perform the final index update operation, thereby further reducing the number of data copies, speeding up the data download speed, reducing CPU consumption, and improving the processing performance of the storage device.

[0190] The above mainly takes the example of the storage device storing the target data in the disk in a logical order as an example. In another optional implementation, the storage device can store the target data in a random order.

[0191] In this implementation, the storage device may store the target data in a disk in a non-logical order.

[0192] Optionally, the storage device may store the data corresponding to all requests included in the target data stored in the second memory in a random order, or may store the data corresponding to only some requests in a random order. The specific setting may be based on actual needs and is not limited here.

[0193] The prerequisite for the storage device to support the out-of-order storage of data corresponding to all requests is that there is no logical relationship between all write requests, and the storage device knows that there is no logical relationship between the multiple requests before storing the target data to the disk.

[0194] The prerequisite for the storage device to support the random downloading of data corresponding to partial requests is that there is no logical relationship between the partial requests included in the target data, and the storage device knows that there is no logical relationship between the multiple partial requests before storing the target data on the disk. Each partial request is still downloaded to the disk in a logical order. For example, the target data includes 200 requests. These 100 requests and another 100 requests may not be downloaded to the disk in a logical order, and the data corresponding to each request included in the two 100 requests are all downloaded to the disk in a logical order. The logical relationship between each request will inform the computing device. When the computing device reads the data later, if it finds that the data has been downloaded from the disk, the computing device can read the data from the disk according to the logical relationship.

[0195] Optionally, after the target data is stored in the disk, the request can be associated with the location of the disk according to a certain correspondence. When the computing device reads the target data in the disk, the target data can be obtained according to the correspondence.

[0196] It is understandable that the embodiments of the present application are mainly described based on the case where the data includes data of multiple write requests. When the target data only includes data of one write request, in order to avoid data errors, the storage device must store the data in a logical order.

[0197] 2. Data reading process

[0198] See also Figure 8 , Figure 8 Another flow chart of the data processing method provided in the embodiment of the present application is shown in FIG. Figure 1The computing device and storage device shown execute, including step 801 to step 802. For the convenience of description, the memory is described as a disk.

[0199] Step 801: A computing device generates a read request for target data.

[0200] Step 802: The computing device obtains target data from the storage device.

[0201] In this embodiment, after generating a read request for the target data, the computing device will first determine whether the target data is in the second memory of the storage device. If so, the target data will be directly read from the second memory of the storage device. If not, it means that the target data has been downloaded from the disk, and the computing device needs to read the target data from the disk.

[0202] After each download of the data in the second memory, the receiving network card will inform the sending network card in real time which data addresses have become invalid and have been downloaded, and then the first network card will feedback to the controller of the computing device. Therefore, the computing side can directly determine whether the target data to be read has been downloaded.

[0203] In an optional implementation, when a first condition is met, the computing device obtains target data according to a first index relationship.

[0204] In this implementation, the first condition is that the computing device determines that the target data is in the second memory according to the first index relationship. After the computing device receives a request to read the target data, if the computing device determines that the first condition is met, the computing device directly obtains the target data from the first memory according to the first index relationship.

[0205] Optionally, if the computing device determines that the first condition is not met, it means that the target data has been downloaded, and the computing device can use the general protocol to send a read request for the target data to the storage device. After the computing device obtains the target data, it clears the address information associated with the target data in the first index relationship.

[0206] It is understandable that in Figure 3a and 3b In the adopted WAL solution, if the computing device determines that the target data is still in the WAL and has not been saved to disk, the computing device will send a read request to the sending network card, the sending network card sends a read request to the second network card, the second network card sends a read request to the WAL, reads the target data from the WAL, and then feeds back. Compared with the aforementioned solution, when the target data is stored in the second memory and has not yet been saved to disk, the embodiment of the present application reduces the number of data copies by directly reading the target data from the second memory that maintains power.

[0207] exist Figures 1 to 8On the basis of the corresponding embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices for implementing the above solutions. Fig. 9 , Fig. 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application.

[0208] The computing device 9000 may be Figure 1 The computing device 9000 includes:

[0209] A generating module 9001 is used to generate a write request for target data;

[0210] The processing module 9002 is used to write the target data into the first memory bound to the sending network card, and obtain the address information of the target data in the second memory, the sending network card sends the target data to the second memory of the storage device in a logical order, the sending network card sends the target data in a remote memory access manner, and the second memory is a power-saving memory;

[0211] The transceiver module 9003 is used to send address information to the storage device, and the address information is used by the storage device to store the target data from the second memory to the storage device in a logical order.

[0212] In a possible implementation, the processing module 9002 is further configured to:

[0213] The sending network card receives the first information sent by the receiving network card, where the first information includes address information of the second memory arranged in a logical order;

[0214] The sending network card sends the target data to the second memory of the receiving network card according to the first information.

[0215] In a possible implementation, the generating module 9001 is further used to generate a read request of the target data;

[0216] The processing module 9002 is further configured to obtain the target data from the second memory when the target data is in the second memory.

[0217] In a possible implementation, the sending network card is a remote direct memory access (RDMA) network card, and the sending network card uses a unilateral RDMA protocol to send target data to the second memory of the storage device.

[0218] In a possible implementation, the sending network card uses the Compute Express Link (CXL) protocol to send the target data to the second memory of the storage device.

[0219] In this embodiment, the operations performed by each unit in the computing device 9000 are the same as those described above. Figure 5 and Figure 8The method described in the illustrated embodiment is similar and can be used to implement the functions of the computing device in the above method embodiment, and can also achieve the beneficial effects possessed by the above method embodiment, which will not be described in detail here.

[0220] See also Fig.10 , Fig.10 A schematic diagram of a storage device provided in an embodiment of the present application. The storage device 1000 may be as follows Figure 1 The storage device 1000 includes:

[0221] The transceiver module 1001 is used to receive the address information of the target data sent by the computing device, the address information is used to indicate the location of the target data in the second memory, the target data is stored in the second memory of the receiving network card in a logical order, the receiving network card receives the target data in a remote memory access manner, and the second memory is a power-saving memory;

[0222] The processing module 1002 is used to store the target data from the second memory to the storage according to the address information.

[0223] In a possible implementation, the processing module 1002 is further configured to:

[0224] The storage device obtains the target data according to the address information;

[0225] The storage device performs a deserialization operation on the target data to obtain the deserialized target data;

[0226] The storage device writes the deserialized target data into the memory.

[0227] In a possible implementation, the processing module 1002 is further configured to:

[0228] The storage device obtains the target data according to the address information;

[0229] The storage device sends data exchange instructions to the second memory and the storage device respectively, so that the data in the second memory and the storage device are exchanged.

[0230] In a possible implementation, the processing module 1002 is further configured to:

[0231] The storage device determines the location of the memory slice where the target data is located in the second memory according to the address information;

[0232] The storage device determines the readable range based on the location of the memory slice;

[0233] The storage device obtains the target data from the readable range in a logical order.

[0234] In a possible implementation, the receiving network card is a single-sided remote direct memory access (RDMA) network card, and the receiving network card uses the RDMA protocol to receive target data sent by the computing device.

[0235] In a possible implementation manner of the fourth aspect, the receiving network card uses the Compute Express Link (CXL) protocol to receive the target data sent by the computing device.

[0236] In this embodiment, the operations performed by each unit in the storage device 1000 are the same as those described above. Figure 5 and Figure 8 The method described in the illustrated embodiment is similar and can be used to implement the functions of the storage device in the above method embodiment, and can also achieve the beneficial effects possessed by the above method embodiment, which will not be described in detail here.

[0237] The following is an exemplary introduction to the hardware structure provided by this application.

[0238] See also Fig.11 , Fig.11 Another structural diagram of a computing device provided in an embodiment of the present application. Fig.11 As shown, computing device 1100 is implemented by a general bus architecture.

[0239] The computing device 1100 includes at least one processor 1101 , a communication bus 1102 , a memory 1103 , and at least one communication interface 1104 .

[0240] The processor 1101, the memory 1103, and the communication interface 1104 communicate via the communication bus 1102, and may also communicate via other means such as wireless transmission. The memory 1103 is used to store instructions, and the processor 1101 is used to execute the instructions stored in the memory 1103. The memory 1103 stores program codes, and the processor 1101 may call the program codes stored in the memory 1103 to execute the steps of the aforementioned method embodiment.

[0241] Optionally, the processor 1101 is a general-purpose central processing unit (CPU), and may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices (PLD), transistor logic devices, hardware components, or any combination thereof. The above-mentioned PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0242] The communication bus 1102 is used to transmit information between the processor 1101, the memory 1103 and the communication interface 1104. The communication bus 1102 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0243] Optionally, the memory 1103 is a read-only memory (ROM) or other types of static storage devices that can store static information and instructions. Alternatively, the memory 1103 is a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. Alternatively, the memory 1103 is an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. Optionally, the memory 1103 exists independently and is connected to the processor 1101 through the communication bus 1102. Optionally, the memory 1103 and the processor 1101 are integrated together.

[0244] The communication interface 1104 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 1104 includes a wired communication interface. Optionally, the communication interface 1104 also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface or a combination thereof, etc. In an embodiment of the present application, the communication interface 1104 includes a sending network card that uses a remote access protocol for communication. For example, an RDMA network card, etc.

[0245] In a specific implementation, as an embodiment, the processor 1101 includes one or more CPUs, such as Fig.11 CPU0 and CPU1 are shown in the figure.

[0246] In a specific implementation, as an embodiment, the computing device 1100 includes multiple processors, such as Fig.11 1 and 1105. Each of these processors is a single-CPU or a multi-CPU. A processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0247] In some embodiments, the memory 1103 is used to store program codes for executing the solution of the present application, and the processor 1101 executes the program codes stored in the memory 1103. That is, the computing device 1100 implements the above method embodiments through the processor 1101 and the program codes in the memory 1103.

[0248] It is to be understood that the method steps in the embodiments of the present application may be implemented in hardware or in software instructions executable by the processor 1101. The software instructions may be composed of corresponding software modules, and the software modules may be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a compact disc read-only memory (CD-ROM) or any other form of storage medium known in the art. In addition, the scope of the device described in the present application is not limited thereto, and the structure of the device may not be limited thereto. Fig.11 A device may be a stand-alone device or may be part of a larger device. For example, a device may be:

[0249] (1) An independent integrated circuit (IC), or chip, or chip system or subsystem;

[0250] (2) having a set of one or more ICs, and optionally, the IC set may also include a storage component for storing data and / or instructions;

[0251] (3) Modules that can be embedded in other devices;

[0252] (4) Receivers, terminals, intelligent terminals, wireless devices, handheld devices, mobile units, vehicle-mounted devices, artificial intelligence devices, machinery and equipment, household equipment, medical equipment, industrial equipment, etc.;

[0253] (5)Others

[0254] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0255] See also Fig.12 , Fig.12 Another structural diagram of a storage device provided in an embodiment of the present application. Fig.12 As shown, the storage device 1200 is implemented by a general bus architecture.

[0256] The storage device 1200 includes at least one processor 1201 , a communication bus 1202 , a memory 1203 , and at least one communication interface 1204 .

[0257] The processor 1201, the memory 1203, and the communication interface 1204 communicate via the communication bus 1202, and may also communicate via other means such as wireless transmission. The memory 1203 is used to store instructions, and the processor 1201 is used to execute the instructions stored in the memory 1203. The memory 1203 stores program codes, and the processor 1201 may call the program codes stored in the memory 1203 to execute the steps of the aforementioned method embodiment.

[0258] Optionally, the processor 1201 is a general-purpose central processing unit (CPU), and may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices (PLD), transistor logic devices, hardware components, or any combination thereof. The above-mentioned PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0259] The communication bus 1202 is used to transmit information between the processor 1201, the memory 1203 and the communication interface 1204. The communication bus 1202 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0260] Optionally, the memory 1203 is a read-only memory (ROM) or other types of static storage devices that can store static information and instructions. Alternatively, the memory 1203 is a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. Alternatively, the memory 1203 is an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited to this. Optionally, the memory 1203 exists independently and is connected to the processor 1201 through the communication bus 1202. Optionally, the memory 1203 and the processor 1201 are integrated together.

[0261] The communication interface 1204 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 1204 includes a wired communication interface. Optionally, the communication interface 1204 also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface or a combination thereof, etc. In an embodiment of the present application, the communication interface 1204 includes a sending network card that uses a remote access protocol for communication. For example, an RDMA network card, etc.

[0262] In a specific implementation, as an embodiment, the processor 1201 includes one or more CPUs, such as Fig.12 CPU0 and CPU1 are shown in the figure.

[0263] In a specific implementation, as an embodiment, the storage device 1200 includes multiple processors, such as Fig.12 1 and 1205. Each of these processors is a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0264] In some embodiments, the memory 1203 is used to store program codes for executing the solution of the present application, and the processor 1201 executes the program codes stored in the memory 1203. In other words, the storage device 1200 implements the above method embodiments through the processor 1201 and the program codes in the memory 1203.

[0265] It is to be understood that the method steps in the embodiments of the present application may be implemented in hardware or in software instructions executable by the processor 1201. The software instructions may be composed of corresponding software modules, and the software modules may be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, mobile hard disks, compact disc read-only memory (CD-ROM) or any other form of storage medium known in the art. In addition, the scope of the device described in the present application is not limited thereto, and the structure of the device may not be limited thereto. Fig.12 A device may be a stand-alone device or may be part of a larger device. For example, a device may be:

[0266] (1) An independent integrated circuit (IC), or chip, or chip system or subsystem;

[0267] (2) having a set of one or more ICs, and optionally, the IC set may also include a storage component for storing data and / or instructions;

[0268] (3) Modules that can be embedded in other devices;

[0269] (4) Receivers, terminals, intelligent terminals, wireless devices, handheld devices, mobile units, vehicle-mounted devices, artificial intelligence devices, machinery and equipment, household equipment, medical equipment, industrial equipment, etc.;

[0270] (5)Others

[0271] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0272] An embodiment of the present application also provides a computer-readable storage medium, including program instructions, which, when executed on a computing device, causes the computing device to execute any of the aforementioned method embodiments.

[0273] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute any of the aforementioned method embodiments.

[0274] It is understood that the devices and methods described in the present application can also be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0275] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0276] The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of the units in this application is a logical division. There may be other division methods when implemented in actual applications. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the unit or subunit described as a separate component may or may not be physically separated, may or may not be a physical unit, or may be distributed in multiple circuit units, and some or all of the units can be selected according to actual needs to achieve the purpose of the present application solution.

[0277] The above is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The various digital numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic.

[0278] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: The computing device generates a write request for target data; The computing device writes the target data into the first memory of the sending network card, and obtains the address information of the target data in the second memory. The sending network card sends the target data to the second memory of the storage device in a logical order. The sending network card sends the target data in a remote memory access manner. The second memory is a power-saving memory. The computing device sends the address information to the storage device, and the address information is used by the storage device to store the target data from the second memory to a storage device according to the logical sequence.

2. The method according to claim 1, characterized in that The sending network card sends the target data to the second memory of the storage device according to the logical order, including: The sending network card receives first information sent by the receiving network card of the storage device, where the first information includes address information of the second memory arranged in the logical order; The sending network card sends the target data to the second memory of the receiving network card according to the first information.

3. The method according to claim 1 or 2, characterized in that: Also includes: The computing device generates a read request for the target data; When the target data is in the second memory, the computing device obtains the target data from the second memory.

4. The method according to any one of claims 1 to 3, characterized in that The sending network card is a remote direct memory access (RDMA) network card, and the sending network card uses a unilateral RDMA protocol to send the target data to the second memory of the storage device.

5. The method according to any one of claims 1 to 3, characterized in that The sending network card sends the target data to the second memory of the storage device using the Compute Express Link (CXL) protocol.

6. A data processing method, characterized in that: include: The storage device receives address information of target data sent by the computing device, the address information is used to indicate the location of the target data in the second memory, the target data is stored in the second memory of the receiving network card in a logical order, the receiving network card receives the target data in a remote memory access manner, and the second memory is a power-saving memory; The storage device stores the target data from the second memory to a storage device according to the address information.

7. The method according to claim 6, characterized in that The storage device stores the target data from the second memory to the storage according to the address information, including: The storage device acquires the target data according to the address information; The storage device performs a deserialization operation on the target data to obtain the deserialized target data; The storage device writes the deserialized target data into the memory.

8. The method according to claim 6, characterized in that The storage device stores the target data from the second memory to the storage according to the address information, including: The storage device acquires the target data according to the address information; The storage device sends data exchange instructions to the second memory and the memory respectively, so that the data in the second memory and the memory are exchanged.

9. The method according to claim 7 or 8, characterized in that: The storage device acquires the target data according to the address information, including: The storage device determines the location of the memory slice where the target data is located in the second memory according to the address information; The storage device determines a readable range according to the location of the memory slice; The storage device acquires the target data from the readable range according to the logical order.

10. The method according to any one of claims 6 to 9, characterized in that The receiving network card is a single-sided remote direct memory access (RDMA) network card, and the receiving network card uses the RDMA protocol to receive the target data sent by the computing device.

11. The method according to any one of claims 6 to 9, characterized in that The receiving network card receives the target data sent by the computing device using the Compute Express Link (CXL) protocol.

12. A computing device, characterized in that: include: A generating module, used for generating a write request for target data; A processing module, used for writing the target data into a first memory bound to a sending network card, and obtaining address information of the target data in a second memory, wherein the sending network card sends the target data to the second memory of the storage device in a logical order, and the sending network card sends the target data in a remote memory access manner, and the second memory is a power-saving memory; The transceiver module is used to send the address information to the storage device, and the address information is used by the storage device to store the target data from the second memory to the storage according to the logical order.

13. A storage device, characterized in that: include: a transceiver module, configured to receive address information of target data sent by a computing device, wherein the address information is used to indicate a location of the target data in the second memory, wherein the target data is stored in the second memory of a receiving network card in a logical order, wherein the receiving network card receives the target data in a remote memory access manner, and wherein the second memory is a power-saving memory; A processing module is used to store the target data from the second memory to a storage device according to the address information.

14. A computing device, characterized in that The method comprises a processor, wherein the processor runs program instructions in the memory to perform the method according to any one of claims 1 to 5.

15. A storage device, characterized in that: The method comprises a processor which runs program instructions in a memory to perform the method according to any one of claims 6 to 11.

16. A storage system, characterized in that: The storage system includes at least one computing device and at least one storage device. The computing device is used to perform the method according to any one of claims 1 to 5; The storage device is used to execute the method according to any one of claims 6 to 11.

Images