Data storage method and device

Abstract

The present disclosure provides a data storage method and device, which includes: receiving a write data request transmitted by a server, the write data request is used to request that target data be stored in a storage cluster, the storage cluster is used to store and manage data to support data writing and reading; storing the target data in a persistent memory in a data processor, and dynamically recording the data storage status in the persistent memory in a constructed index table, the index table supports the server to read all or part of the target data; reading the target data from the persistent memory in sequence and writing it into the storage cluster until all the target data indicated by the write data request is written. The present disclosure uses persistent memory to divide data storage into two stages, the first stage is to store the target data in the persistent memory, and the second stage is to take the target data out of the persistent memory and store it in the storage cluster, which can achieve efficient storage of the target data.

Description

Technical Field

[0001] The present disclosure relates to the field of data storage, and more particularly to a data storage method and device. Background Art

[0002] Currently, when write data needs to be stored in a storage cluster, a data processor (DPU, Data Processing Unit) is required to send the write data to the central node of the storage cluster through the network. The central node then distributes the write data to different storage nodes through internal logic to complete the storage of multiple copies. After the storage node completes the storage, it needs to feedback the stored information to the central node, and then the central node feedbacks the stored information to the data processor. The data processor then feedbacks the stored information to the server. The server can only store the next write data after receiving the stored information. The entire storage process is lengthy and complicated, resulting in delays in storing write data.

[0003] In summary, there is an urgent need for a method to reduce the latency of storing and writing data. Summary of the Invention

[0004] The embodiments of the present disclosure provide a data storage method and device to reduce the storage data writing delay.

[0005] In a first aspect, an embodiment of the present disclosure provides a data storage method, which receives a write data request transmitted by a server, where the write data request is used to request that target data be stored in a storage cluster, where the storage cluster is used to store and manage data to support data writing and reading; the target data is stored in a persistent memory in a data processor, and the data storage status in the persistent memory is dynamically recorded in a constructed index table, where the index table supports the server to read all or part of the target data; the target data is read from the persistent memory in sequence and written into the storage cluster until all the target data indicated by the write data request is written.

[0006] In a second aspect, an embodiment of the present disclosure provides a data storage device, including:

[0007] A receiving unit, configured to receive a write data request transmitted by a server, wherein the write data request is used to request that target data be stored in a storage cluster, and the storage cluster is used to store and manage data to support data writing and reading;

[0008] A storage unit is used to store the target data in the persistent memory of the data processor and dynamically record the data storage status in the persistent memory in a constructed index table. The index table supports the server to read all or part of the target data;

[0009] The reading unit is used to read the target data from the persistent memory in sequence and write it into the storage cluster until all the target data indicated by the write data request is written.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0011] Memory stores computer-executable instructions;

[0012] At least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data storage method provided in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the data storage method provided in the first aspect above is implemented.

[0014] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided. The computer program product includes computer-executable instructions. When a processor executes the computer-executable instructions, the data storage method provided in the first aspect above is implemented.

[0015] The data storage method and device provided in this embodiment include: receiving a write data request transmitted by a server, the write data request is used to request that target data be stored in a storage cluster, the storage cluster is used to store and manage data to support data writing and reading; storing the target data in a persistent memory in a data processor, and dynamically recording the data storage status in the persistent memory in a constructed index table, the index table supports the server to read all or part of the target data; reading the target data from the persistent memory in sequence and writing it to the storage cluster until all the target data indicated by the write data request is written. The present disclosure uses persistent memory to divide data storage into two stages, the first stage is to store the target data in the persistent memory, and the second stage is to take the target data out of the persistent memory and store it in the storage cluster, which can achieve efficient storage of the target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A schematic diagram of storing data provided for related technologies;

[0018] Figure 2 A schematic diagram of storing data provided in an embodiment of the present disclosure;

[0019] Figure 3 A flowchart of a data storage method according to an embodiment of the present disclosure;

[0020] Figure 4 A flowchart of another data storage method provided by an embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram of the structure of a persistent memory provided by an embodiment of the present disclosure;

[0022] Figure 6 A structural block diagram of a data storage device provided in an embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0025] In the related art, the Figure 1 The data is stored in the way of Figure 1 , including: servers, DPUs and storage clusters, where the storage cluster can include at least one central node and N storage nodes, where n is a positive integer. When the server stores data, the solid line in the figure is the path for storing the target data, and the dotted line is the path for returning feedback information after the data is stored. The sum of these two paths is the complete path for a completed data storage. When there are multiple data, the previous data can only continue to store the next data after it has completed a complete path. It can be seen that this storage method needs to take into account some factors that affect latency, such as the stability of the network environment between the data processor and the storage cluster, the execution efficiency of the multiple storage nodes within the storage cluster, and the latency introduced by the network environment within the storage cluster. In addition, a series of tasks such as data recovery, data compression, EC encoding, multi-node requests, and data balancing in the storage cluster lead to competition for resources when writing data, including but not limited to: CPU resources, network bandwidth resources, data disk SSD resources, etc. These influencing factors will lead to storage latency problems.

[0026] Based on the above problems, the data storage method provided by the present disclosure is to use Figure 2The storage of target data is divided into two stages. The first stage is to store the target data in the DPU's persistent memory (PMEM), and then the DPU returns the stored information to the server. The second stage is to obtain the target data from the persistent memory and store it in the storage cluster. The two stages are performed in parallel. Therefore, when there are multiple target data, the storage efficiency of write data can be improved and the storage latency can be reduced.

[0027] refer to Figure 3 , is a flow chart of a data storage method provided by an embodiment of the present disclosure. The data storage method is applied to Figure 2 The data processor shown in the figure, the data storage method specifically includes the following steps:

[0028] S301: Receive a write data request transmitted by a server.

[0029] The write data request is used to request that target data be stored in a storage cluster, and the storage cluster is used to store and manage data to support data writing and reading.

[0030] In the embodiment of the present disclosure, the target data is data of an integer multiple of 4KB, and includes metadata and data.

[0031] The server needs to store multiple target data, and the target data may be one of the multiple target data.

[0032] Furthermore, the data processor can realize storage offloading, that is, the bare metal server or virtual machine server does not need to have a local disk on the server, and all interactions with the cloud disk (that is, the storage cluster) are transferred to the data processor for completion.

[0033] S302: Storing the target data in the persistent memory of the data processor, and dynamically recording the data storage status in the persistent memory in the constructed index table.

[0034] The index table supports the server to read all or part of the target data;

[0035] In the present disclosure, an index table is set in a data processor, wherein when the index table is initially constructed, any table structure among a red-black tree, a dictionary tree or a key-value pair is selected to construct the index table.

[0036] In addition, the index table can also use other commonly used key-value indexing methods. The key can be the identifier of the written data, and the value can be the storage location of the written data.

[0037] The index table can satisfy the server's potential read requests for data in persistent memory. If the latency in the second phase is higher than in the first phase, there may be a backlog of write data in persistent memory for a short period of time. This allows the server to read the write data in persistent memory, alleviating the backlog.

[0038] In the disclosed embodiments, persistent memory is a solid-state, high-performance, byte-addressable memory device residing on a memory bus. Located on the memory bus, persistent memory supports data access similar to DRAM (dynamic random access memory), offering comparable speed and latency to DRAM, combined with the non-volatility of NAND (a type of flash memory).

[0039] In the embodiment of the present disclosure, S301 to S302 completes the first stage transmission of the target data.

[0040] S303: Read the target data from the persistent memory in sequence and write them into the storage cluster until all the target data indicated by the data write request is written.

[0041] Before S303 , the method further includes: feeding back stored information of the target data to the server, where the stored information is used to instruct the server to transmit the next write data to the data processor.

[0042] Among them, after the target data is stored in the persistent memory, it can be considered that the target data has been stored, and the stored information of the target data can be fed back to the server, so that the server can transmit the next target data to the data processor without waiting until the target data has been stored in the storage cluster before transmitting the next target data, which can improve the storage efficiency of data on the server side.

[0043] Furthermore, the target data can be written to the storage cluster later, completing the second stage of the target data.

[0044] In addition, refer to Figure 3 After the target data is sent to the storage cluster for storage based on MAC through the network, feedback information returned by the storage cluster through the network can be received to indicate that the target data has been written to the storage cluster. The data processor can then modify the state of the target data in the persistent memory.

[0045] In an optional embodiment, the target data written to the storage cluster may also be deleted from the persistent memory to release storage space of the persistent memory.

[0046] In this disclosure, a "two-stage write solution" is adopted by dividing the write data path stored on the DPU side: the lengthy and complex (various contentions and instabilities) write data path that interacts with the cluster is shielded from the host side and hosted by the "second stage" DPU.

[0047] refer to Figure 4 , is a flow chart of another data storage method provided by an embodiment of the present disclosure. Figure 4 As shown, the data storage method specifically includes the following steps:

[0048] S401: Receive a write data request transmitted by a server.

[0049] The implementation process of this step refers to S301 and will not be repeated here.

[0050] S402 : Search for a free storage area in the persistent memory, and write the target data into the free storage unit according to the circular order of the circular buffer.

[0051] The persistent memory uses a first-in-first-out circular buffer structure. The persistent memory includes multiple storage units, and the target data is written to the idle storage units.

[0052] Specifically, the persistent memory includes multiple sequentially connected storage units. Each storage unit includes a metadata header area and a data body area. The metadata header area is used to store metadata of the target data, and the data body area is used to store the target data. In addition, the metadata header area also includes a metadata flag bit, which is used to dynamically mark the data storage status.

[0053] Reference Figure 5 , the persistent memory includes m storage units (a1 to am), m is a positive integer, and the storage unit ai (i is 1 to m) includes: metadata header area bi and data body area ci.

[0054] In the present disclosure, the order of storing write data may be a1, a2, ..., am, and the order of reading write data may also be a1, a2, ..., am.

[0055] For example, if all storage units in the persistent memory are idle, the first target data is stored in storage unit a1, the second target data is stored in storage unit a2, and the mth target data is stored in storage unit am. During this period, if the first target data in storage unit a1 through the xth target data in storage unit ax (x is an integer greater than 1 and less than m) have been sent to the storage cluster, then storage units a1 through ax release their respective target data. The m+1th target data is then stored in storage unit a1, and the m+2th target data is stored in storage unit a2. This cycle continues, implementing cyclic storage in persistent memory.

[0056] In the present disclosure, at least one byte may be reserved in the metadata header area as a metadata mark bit to dynamically mark the data storage status, specifically to record the status of the storage unit and / or the target data in the storage unit.

[0057] Among them, the metadata flag bit supports three states of data, including: the first state is used to indicate that the data is not stored in the persistent memory, the second state is used to indicate that the data has been written to the persistent memory but not written to the storage cluster, and the third state is used to indicate that the data has been written to the storage cluster. If the first state is represented by 0, the data is not stored in the persistent memory, which means that the storage unit is in an idle state. The second state is represented by 1, and the data has been written to the persistent memory but not written to the storage cluster, which means that the corresponding storage unit has stored the target data, but the target data has not been stored in the storage cluster. The third state is represented by 2, and the data has been written to the storage cluster, which means that the corresponding storage unit has stored the target data, and the target data has been stored in the storage cluster.

[0058] In the embodiment of the present disclosure, after determining that the metadata mark bit is in the third state, the target data of the storage unit where the metadata mark bit is located can be deleted to release the storage unit.

[0059] Further, in the present disclosure, the target data is written into the free storage unit in the circular order of the circular buffer, for example, referring to Figure 5 If the target data is stored in the storage unit ay (y is an integer greater than 1 and less than m), and the storage units az (z is an integer greater than y) after ay do not store the target data, the target data can be stored in the storage unit a(y+1).

[0060] In an embodiment of the present disclosure, if the target data is written to an idle storage unit, the metadata mark bit of the idle storage unit is modified from a first state to a second state, wherein the first state indicates that the target storage unit is in an idle state, and the second state indicates that the target data body area has stored the target data, and the target data has not been stored in the storage cluster.

[0061] S403: Read the target data from the persistent memory in sequence and write them into the storage cluster until all the target data indicated by the data write request is written.

[0062] For example, referring to Figure 5 If the target data is stored in storage unit ay, and the target data in storage unit ap (p is less than y) before storage unit ay is sent to the storage cluster for storage, the target data is obtained in storage unit ay and sent to the storage cluster for storage.

[0063] Furthermore, if the target data of one of the storage units is stored in the storage cluster, the metadata flag of the storage unit is updated from the second state to the third state, wherein the third state indicates that the target data in the storage unit has been stored in the storage cluster.

[0064] In an embodiment of the present disclosure, when the data processor is powered off and restarted, the data storage method also includes: obtaining the metadata mark bit of the storage unit in the persistent memory; if the metadata mark bit is marked as the second state, continuing to read the target data from the storage unit and write it into the storage cluster.

[0065] Among them, persistent memory has the characteristic of persistence. When the server is powered off or the data processor is powered off and restarted, it can ensure that data will not be lost and continue to interact with the storage cluster.

[0066] After power failure and restart, the metadata flag bit of each storage unit can be read. If the metadata flag bit is marked as the second state, the target data will continue to be read from the storage unit and written into the storage cluster in the cyclic order of the cyclic buffer area until all the target data indicated by the write data request is written.

[0067] Furthermore, when the write throughput supported by the server is less than a preset threshold, the target data is stored in the persistent memory of the data processor.

[0068] Among them, when the write throughput supported by the server is less than the preset threshold, the disclosed solution is adopted; otherwise, the Figure 1 The scheme shown in FIG. 1 , wherein the preset threshold is set in advance according to the size of the persistent memory.

[0069] In addition, persistent memory is byte-addressable and has low latency at the nanosecond level, which can significantly reduce write latency.

[0070] Furthermore, if the first phase ends, meaning the server no longer writes target data to persistent memory, the second phase can continue, aiming to complete the process of writing data from persistent memory to the storage cluster within a specified timeframe, until all data in persistent memory has been written to the storage cluster. The specified timeframe can be estimated based on the size of the persistent memory and the write latency of the second phase; this disclosure does not limit the specific estimation method.

[0071] Specifically, if the server immediately shuts down the cloud disk (the cloud disk includes the process of sending the target data from the server to the DPU, and then from the DPU to the storage cluster), the cloud disk shutdown is also divided into two phases. The first phase is that the server can quickly shut down the cloud disk. That is, from the server side, the cloud disk has been shut down; from the perspective of the data processor, this means that the "first phase write" has exited, and the "first phase write" shutdown is completed. The other phase is that after the server shuts down the cloud disk, the data processor continues the "second phase write" until all the corresponding data in the persistent memory has been written to the storage cluster. The data processor executes the second phase of the cloud disk shutdown operation.

[0072] In addition, the data processor can maintain a variable, such as "data_in_pmem_count," for each cloud disk using persistent memory. The data processor periodically synchronizes the value (count) in this variable to the storage cluster (one of the management nodes). For example, if the value is non-zero, it indicates that the cloud disk is still in use. If the value is 0, it indicates that the cloud disk is closed.

[0073] In another optional embodiment, if it is necessary to close the cloud disk on the current server and then open the cloud disk on the new server, when opening the cloud disk on the new server, the storage cluster side performs the following optional support schemes: one scheme is that the storage cluster blocks the new server's request to open the cloud disk until the data processor closes the cloud disk in the "second phase" corresponding to the old server. Another scheme is that the storage cluster side supports the new server to directly open the cloud disk, and can continue to complete the writing of the target data corresponding to part of the old server in the persistent memory, wherein the target data of the new server in PMEM has not been fully synchronized to the storage cluster.

[0074] In an embodiment of the present disclosure, a "extreme write latency" offload solution is provided through a server's cloud disk, meeting data write requirements in various scenarios. This application embeds PMEM into the DPU, splitting the write data path for "highly write latency-sensitive cloud disks" into a two-stage process. This leverages PMEM's byte-addressability and persistence to ensure that data written by the server remains accessible after a DPU power failure and restart. Furthermore, when the target data volume is small and PMEM has ample storage space, a "read cache" solution can be further developed to reduce read latency. Furthermore, this disclosure leverages PMEM's persistence and byte-addressability to provide a data recovery method after a DPU power failure and restart (or after a process exits and restarts). When the server shuts down the cloud disk, this application asynchronously handles the interaction between the "second-stage write" and the cluster, ensuring instant, non-blocking shutdown of the cloud disk. Furthermore, this disclosure can simplify the server-side write request data path, providing a stable and extreme write latency experience for the cloud disk.

[0075] In adopting Figure 1 With related technologies, a single-path write latency of 4KB can reach 200-300µs. By adopting the disclosed embodiments, most of this latency is hidden, reducing the ultimate write latency to 20-30µs. Furthermore, latency is closed-loop across the server-side storage stack, the DPU link, and the processing efficiency of the DPU-to-PMEM interface, eliminating the impact of network and storage cluster constraints.

[0076] Corresponding to the data storage method of the above embodiment, Figure 61 is a structural block diagram of a data storage device 60 provided in an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 6 As shown, the data storage device includes: a receiving unit 601, a storage unit 602, and a reading unit 603, wherein:

[0077] A receiving unit 601 is configured to receive a write data request transmitted by a server, where the write data request is used to request that target data be stored in a storage cluster, and the storage cluster is configured to store and manage data to support writing and reading of data;

[0078] Storage unit 602, used to store target data in the persistent memory of the data processor and dynamically record the data storage status in the persistent memory in the constructed index table. The index table supports the server to read all or part of the target data;

[0079] The reading unit 603 is configured to sequentially read target data from the persistent memory and write the target data into the storage cluster until all the target data indicated by the data write request is written.

[0080] In some embodiments, the persistent memory adopts a first-in-first-out circular buffer structure, and the storage unit 602 is specifically used to: search for a free storage area of ​​the persistent memory and write the target data into the free storage unit according to the circular order of the circular buffer.

[0081] In some embodiments, the persistent memory is a circular buffer and further includes: a selection unit (not shown) for selecting any table structure of a red-black tree, a dictionary tree, or a key-value pair to construct the index table when the index table is initially constructed.

[0082] In some embodiments, when the write throughput supported by the server is less than a preset threshold, storing the target data into the persistent memory in the data processor is performed.

[0083] In some embodiments, the persistent memory includes a plurality of storage units connected sequentially, each storage unit including a metadata header area and a data body area, the metadata header area is used to store metadata, and the data body area is used to store data.

[0084] In some embodiments, the metadata header area further includes a metadata mark bit, which is used to dynamically mark the data storage status.

[0085] In some embodiments, the metadata flag bit supports three states of data, including: a first state is used to indicate that the data is not stored in the persistent memory, a second state is used to indicate that the data has been written to the persistent memory but not written to the storage cluster, and a third state is used to indicate that the data has been written to the storage cluster.

[0086] In some embodiments, if the data processor is powered off and restarted, the reading unit 603 is also used to obtain the metadata flag bit of the storage unit in the persistent memory; if the metadata flag bit is marked as the second state, the target data continues to be read from the storage unit and written into the storage cluster.

[0087] The data storage device provided in this embodiment can be used to execute the technical solution of the embodiment of the above-mentioned data storage method. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0088] refer to Figure 7 , which shows a schematic structural diagram of an electronic device 70 suitable for implementing an embodiment of the present disclosure. The electronic device 70 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 7 As shown, electronic device 70 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 71, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 72 or programs loaded from storage device 78 into random access memory (RAM) 73. RAM 73 also stores various programs and data required for the operation of electronic device 70. Processing device 71, ROM 72, and RAM 73 are interconnected via a bus 74. An input / output (I / O) interface 75 is also connected to bus 74.

[0090] Typically, the following devices may be connected to the I / O interface 75: an input device 76 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 77 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 78 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 79. The communication device 79 may allow the electronic device 70 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 70 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 79, or installed from the storage device 78, or installed from the ROM 72. When the computer program is executed by the processing device 71, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0092] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0093] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0094] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0095] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a unit, program segment or a part of code, and the unit, program segment or a part of code contains one or more executable instructions for realizing the prescribed logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0097] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0098] The functions described above may be at least partially performed by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), System on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] In a first aspect, according to one or more embodiments of the present disclosure, a data storage method is provided, comprising: receiving a write data request transmitted by a server, the write data request being used to request that target data be stored in a storage cluster, the storage cluster being used to store and manage data to support data writing and reading; storing the target data in a persistent memory in a data processor, and dynamically recording the data storage status in the persistent memory in a constructed index table, the index table supporting the server to read all or part of the target data; reading the target data from the persistent memory in sequence and writing it into the storage cluster until all the target data indicated by the write data request is written.

[0101] In one or more embodiments of the present disclosure, the persistent memory adopts a first-in-first-out circular buffer structure, and storing target data in the persistent memory of the data processor includes:

[0102] Find the free storage area of ​​the persistent memory and write the target data into the free storage unit according to the circular order of the circular buffer area.

[0103] In one or more embodiments of the present disclosure, the method further includes:

[0104] When constructing the index table for the first time, select any table structure among red-black tree, dictionary tree or key-value pair to construct the index table.

[0105] One or more embodiments of the present disclosure execute storing target data into a persistent memory in a data processor when the write throughput supported by the server is less than a preset threshold.

[0106] In one or more embodiments of the present disclosure, the persistent memory includes a plurality of sequentially connected storage units, each storage unit including a metadata header area and a data body area, the metadata header area is used to store metadata, and the data body area is used to store data.

[0107] In one or more embodiments of the present disclosure, the metadata header area further includes a metadata mark bit, which is used to dynamically mark the data storage status.

[0108] In one or more embodiments of the present disclosure, the metadata flag bit supports three states of data, including: a first state is used to indicate that the data is not stored in the persistent memory, a second state is used to indicate that the data has been written to the persistent memory but not written to the storage cluster, and a third state is used to indicate that the data has been written to the storage cluster.

[0109] In one or more embodiments of the present disclosure, when a data processor is powered off and restarted, the data storage method further includes:

[0110] Get the metadata flag of the storage unit in persistent memory;

[0111] If the metadata flag is marked as the second state, the target data continues to be read from the storage unit and written into the storage cluster.

[0112] In a second aspect, according to one or more embodiments of the present disclosure, there is provided a data storage device, comprising:

[0113] A receiving unit, configured to receive a write data request transmitted by a server, wherein the write data request is used to request that target data be stored in a storage cluster, and the storage cluster is used to store and manage data to support data writing and reading;

[0114] A storage unit is used to store the target data in the persistent memory of the data processor and dynamically record the data storage status in the persistent memory in a constructed index table. The index table supports the server to read all or part of the target data;

[0115] The reading unit is used to read the target data from the persistent memory in sequence and write it into the storage cluster until all the target data indicated by the write data request is written.

[0116] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0117] Memory stores computer-executable instructions;

[0118] At least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data storage method provided in the first aspect above.

[0119] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the data storage method provided in the first aspect above is implemented.

[0120] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided. The computer program product includes computer-executable instructions. When a processor executes the computer-executable instructions, the data storage method provided in the first aspect above is implemented.

[0121] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0122] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0123] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A data storage method, characterized in that: include: Receive a write data request transmitted by a server, wherein the write data request is used to request that target data be stored in a storage cluster, wherein the storage cluster is used to store and manage data to support writing and reading of data; Storing the target data in a persistent memory in a data processor, and dynamically recording the data storage status in the persistent memory in a constructed index table, wherein the index table supports the server to read all or part of the target data; Feedback stored information of the target data to the server, where the stored information is used to instruct the server to transmit next write data to the data processor; The target data are sequentially read from the persistent memory and written into the storage cluster until all the target data indicated by the write data request are written; wherein storing the target data into the persistent memory of the data processor and sequentially reading the target data from the persistent memory and writing the target data into the storage cluster are performed in parallel; as well as The persistent memory includes a plurality of storage units connected in sequence, each storage unit includes a metadata header area and a data body area, the metadata header area is used to store metadata, and the data body area is used to store data; the metadata header area also includes a metadata flag bit, and the metadata flag bit is used to dynamically mark the data storage status.

2. The data storage method according to claim 1, wherein: The persistent memory adopts a first-in-first-out circular buffer structure, and storing the target data in the persistent memory in the data processor includes: Search for a free storage area of ​​the persistent memory, and write the target data into the free storage unit according to the circular order of the circular buffer area.

3. The data storage method according to claim 1, wherein: The method further comprises: When constructing the index table for the first time, select any table structure among red-black tree, dictionary tree or key-value pair to construct the index table.

4. The data storage method according to claim 1, wherein: When the write throughput supported by the server is less than a preset threshold, storing the target data in the persistent memory of the data processor is performed.

5. The data storage method according to claim 1, wherein: The metadata flag bit supports three states of data, including: a first state for indicating that the data is not stored in the persistent memory, a second state for indicating that the data has been written to the persistent memory but not written to the storage cluster, and a third state for indicating that the data has been written to the storage cluster.

6. The data storage method according to claim 5, characterized in that: In the case where the data processor is powered off and restarted, the data storage method further includes: Obtaining a metadata flag bit of a storage unit in the persistent memory; If the metadata flag is marked as the second state, the target data is continued to be read from the storage unit and written into the storage cluster.

7. A data storage device, characterized in that include: A receiving unit, configured to receive a write data request transmitted by a server, wherein the write data request is used to request that target data be stored in a storage cluster, wherein the storage cluster is used to store and manage data to support writing and reading of data; a storage unit, configured to store the target data in a persistent memory in the data processor, dynamically record data storage status in the persistent memory in a constructed index table, and feedback stored information of the target data to the server, wherein the stored information is used to instruct the server to transmit the next write data to the data processor, and the index table supports the server in reading all or part of the target data; a reading unit, configured to sequentially read the target data from the persistent memory and write the target data into the storage cluster until all the target data indicated by the write data request is written; wherein storing the target data into the persistent memory of the data processor and sequentially reading the target data from the persistent memory and writing the target data into the storage cluster are performed in parallel; as well as The persistent memory includes a plurality of storage units connected in sequence, each storage unit includes a metadata header area and a data body area, the metadata header area is used to store metadata, and the data body area is used to store data; the metadata header area also includes a metadata flag bit, and the metadata flag bit is used to dynamically mark the data storage status.

8. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data storage method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the data storage method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 6 is implemented.

Images