Data writing method and system, storage medium, electronic device and computer program product

Abstract

The present application discloses a data writing method and system, a storage medium, an electronic device and a computer program product, which relate to the field of storage systems, including: performing encoding calculation on a first data stream by a first processing unit in a storage client, and storing the first data stream after encoding calculation in a cache unit in the first processing unit; and then, according to the first address of the first solid-state hard disk to which the first data stream is to be written and the first identifier of the second address of the cache unit where the first data stream is located, the first storage node including the first solid-state hard disk can write the first data stream after encoding calculation to the first solid-state hard disk. In other words, the embodiment of the present application performs the encoding calculation task in data stream processing by a storage client instead of a CPU, thereby solving the problem of memory resource preemption caused by the CPU running software to execute the encoding calculation task in data stream processing in the related art, and the CPU also needs to process other services when processing the encoding calculation.

Description

Technical Field

[0001] The present application relates to the field of storage systems, and in particular to a data writing method and system, a storage medium, an electronic device, and a computer program product. Background Art

[0002] Distributed storage often employs multi-replica backup mechanisms and erasure coding to ensure data reliability. Because erasure coding involves extensive matrix operations and cross-node data transmission, it imposes significant computational and transmission overhead on the system. However, the current mainstream approach utilizes the CPU to run software to perform erasure encoding and decoding calculations, relying on the computing power of the Central Processing Unit (CPU) rather than dedicated erasure hardware accelerators. When the system's input / output (I / O) bandwidth is high, erasure calculations consume significant CPU resources. Simultaneously, the CPU must also handle tasks from other software modules. The resulting CPU and memory usage inevitably impacts overall performance.

[0003] Therefore, in the related art, when the CPU runs software to execute encoding operation tasks in data stream processing, the CPU also needs to process other services while processing encoding operations, resulting in the preemption of memory resources, which has not yet been effectively solved. Summary of the Invention

[0004] The present application provides a data writing method and system, a storage medium, an electronic device, and a computer program product to at least solve the problem of memory resource preemption caused by the CPU running software to perform encoding operation tasks in data stream processing in the related art. When the CPU processes the encoding operation, it also needs to process other services.

[0005] The present application provides a data writing method, which is applied to a storage client, comprising: upon receiving a data write request, performing a first calculation type of encoding calculation on a first data stream corresponding to the data write request by a first processing unit in the storage client, and storing the encoded first data stream in a second address space of a cache unit in the first processing unit, wherein the first calculation type includes at least one of the following: data erasure, data encryption, and the first calculation type is the calculation type of the first data stream; constructing a write control message according to a first address of a first solid-state hard disk corresponding to the first data stream and a first identifier of a second address corresponding to the second address space, wherein the first address is an address corresponding to a first address space in the first solid-state hard disk for writing the encoded first data stream; sending the write control message to a first storage node including the first solid-state hard disk, so that the first storage node obtains the encoded first data stream according to the write control message, and writes the encoded first data stream to the first solid-state hard disk.

[0006] The present application also provides a data writing system, comprising: a storage client and a target storage node, wherein: the storage client is used to, upon receiving a data write request, perform a first calculation type of decoding calculation on the first data stream corresponding to the data write request through the first processing unit in the storage client, and store the decoded first data stream in the second address space of the cache unit in the first processing unit; construct a write control message according to the first address of the first solid-state hard disk corresponding to the first data stream and the first identifier of the second address corresponding to the second address space, and send the write control message to the first storage node containing the first solid-state hard disk, wherein the first calculation type includes at least one of the following: data erasure, data encryption, the first calculation type is the calculation type of the first data stream, and the first address is the address corresponding to the first address space in the first solid-state hard disk for writing the decoded first data stream; the target storage node comprises: the first storage node is used to obtain the decoded first data stream according to the write control message, and write the decoded first data stream to the first solid-state hard disk.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data writing methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data writing methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data writing methods when executed by a processor.

[0010] Through this application, since the first processing unit in the storage client performs encoding calculation on the first data stream, and stores the first data stream after encoding calculation in the cache unit in the first processing unit; then, according to the first address of the first solid-state hard disk to which the first data stream is to be written and the first identifier of the second address of the cache unit where the first data stream is located, the first storage node containing the first solid-state hard disk can write the first data stream after encoding calculation to the first solid-state hard disk. In other words, the embodiment of the present application performs the encoding calculation task in data stream processing by the storage client instead of the CPU, thereby solving the problem of memory resource preemption caused by the CPU running software to execute the encoding operation task in data stream processing in the related art. When the CPU processes the encoding operation, it also needs to process other services, thereby avoiding the memory resource preemption problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a hardware structure block diagram of a computer terminal for a data writing method according to an embodiment of the present application;

[0013] Figure 2 is a flow chart of a data writing method according to an embodiment of the present application;

[0014] Figure 3 This is a flow chart of a method for performing data stream processing on a PCIE expansion card in the related art;

[0015] Figure 4 This is an architectural diagram corresponding to a write processing flow according to an optional embodiment of the present application;

[0016] Figure 5 is a flowchart of a write data process according to an optional embodiment of the present application;

[0017] Figure 6 is an architectural diagram corresponding to a read processing flow according to an optional embodiment of the present application;

[0018] Figure 7 is a flowchart of a read data process according to an optional embodiment of the present application;

[0019] Figure 8 2 is a framework diagram of a data writing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0022] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0023] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data writing method depends, the specific application environment architecture or specific hardware architecture is described here.

[0024] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure diagram of a computer terminal of a data writing method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0025] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the interactive state in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0026] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0027] The embodiments of the present application provide a data writing method. The following is an explanation of the technical terms involved in the embodiments of the present application:

[0028] Data erasure correction (or erasure coding), where erasure coding (EC) can split data into fragments, expand and encode redundant data blocks, and store them in different locations, such as disks, storage nodes, or other geographical locations.

[0029] Figure 2 This is a flow chart of a data writing method according to an embodiment of the present application, which can be applied to Figure 1 In a computer terminal, such as Figure 2 As shown, the process includes the following steps:

[0030] Step S202: Upon receiving a data write request, performing encoding calculation of a first calculation type on a first data stream corresponding to the data write request by a first processing unit in the storage client, and storing the encoded first data stream in a second address space of a cache unit in the first processing unit, wherein the first calculation type includes at least one of the following: data erasure and data encryption, and the first calculation type is a calculation type of the first data stream;

[0031] The first processing unit may be a data processing unit (DPU) in the storage client, a smart network card in the storage client, or a combination of a DPU and a smart network card in the storage system, expressed as DPU / smart network card.

[0032] Step S204: constructing a write control message based on a first address of the first solid-state drive corresponding to the first data stream and a first identifier of a second address corresponding to the second address space, wherein the first address is an address corresponding to a first address space in the first solid-state drive for writing the first data stream after the encoded calculation;

[0033] Step S206, sending the write control message to the first storage node containing the first solid state drive, so that the first storage node obtains the first data stream after the encoding calculation according to the write control message, and writes the first data stream after the encoding calculation to the first solid state drive.

[0034] According to the data writing method of the present application, the first data stream is encoded and calculated by the first processing unit in the storage client, and the first data stream after encoding and calculation is stored in the cache unit in the first processing unit; then, according to the first address of the first solid-state hard disk to which the first data stream is to be written and the first identifier of the second address of the cache unit where the first data stream is located, the first storage node including the first solid-state hard disk can write the first data stream after encoding and calculation to the first solid-state hard disk. In other words, the embodiment of the present application performs the encoding and calculation tasks in the data stream processing by the storage client instead of the CPU, thereby solving the problem of memory resource preemption caused by the CPU running software to execute the encoding and calculation tasks in the data stream processing in the related art. When the CPU processes the encoding and calculation, it also needs to process other services, thereby avoiding the memory resource preemption problem.

[0035] Optionally, before constructing a write control message according to the first address of the first solid-state hard disk corresponding to the first data stream and the first identifier of the second address corresponding to the second address space in the above-mentioned step S204, the method further includes: obtaining the second data stream corresponding to the data write request, and performing data preprocessing operations on the second data stream to obtain the first data stream; determining whether there is a reserved address space corresponding to the first data stream in multiple solid-state hard disks according to a distributed data layout algorithm, wherein each storage node includes a solid-state hard disk, and multiple storage nodes include: the first storage node, the reserved address space is an address space reserved for writing the first data stream after the encoding calculation; in determining whether there is a reserved address space corresponding to the first data stream in the multiple solid-state hard disks In a case where the reserved address space exists in one or more second solid-state hard drives, the addresses corresponding to the one or more second solid-state hard drives are determined as the first address, wherein the first solid-state hard drive includes: the one or more second solid-state hard drives; in a case where it is determined that the reserved address space does not exist in the multiple solid-state hard drives, consistency negotiation is performed with the multiple storage nodes so that the multiple storage nodes select one or more third solid-state hard drives from the multiple solid-state hard drives and allocate address space for the first data stream in the one or more third solid-state hard drives; the addresses corresponding to the one or more third solid-state hard drives are determined as the first address, wherein the first solid-state hard drive includes: the one or more third solid-state hard drives.

[0036] It is understandable that before constructing the write control message, it is necessary to obtain the first data stream and determine the first address of the address space in the first solid-state drive where the first data stream is to be stored. Specifically:

[0037] When a storage client receives a data write request, it first processes the request to form a second data stream. This second data stream is the original, unprocessed data stream. The storage client then performs preprocessing on the second data stream, such as data striping and metadata processing, to produce the first data stream. The first data stream is the optimized data stream that has undergone preliminary processing and is ready for erasure coding (i.e., erasure coding calculations) or encryption (i.e., encryption calculations).

[0038] Using a distributed data layout algorithm, multiple solid-state drives (SSDs) are checked for address space reserved for the first data stream. Each storage node contains an SSD, and there may be multiple storage nodes. Reserved address space is pre-prepared space for storing encoded data to reduce latency when writing data.

[0039] If no reserved address space is found on the SSD, the storage system client initiates a consistency negotiation with multiple storage nodes. This negotiation determines which SSD (or SSDs) within the storage node can allocate address space for the first data stream. This ensures that all participating nodes agree on the storage location for the first data stream, avoiding conflicts and inconsistencies during data writes.

[0040] After consistency negotiation, the address corresponding to the determined first solid-state hard disk is determined as the first address.

[0041] Optionally, in step S202, the first calculation type in the encoding calculation of the first calculation type performed by the first processing unit in the storage client on the first data stream corresponding to the data write request may include data erasure, data encryption, data deduplication, and other situations. The following is a process of performing the encoding calculation when it is determined that the first calculation type is data erasure and data encryption:

[0042] (1) When it is determined that the first calculation type is data erasure correction, calling a data erasure correction calculation unit in the first processing unit through the first processing unit, and encoding the first data stream according to the data erasure correction calculation unit to generate first erasure correction data and first verification data corresponding to the first data stream; and generating the first data stream after the encoding calculation according to the first erasure correction data and the first verification data.

[0043] It is understandable that the coding calculation of data erasure correction can be implemented by the first processing unit, and the specific steps are:

[0044] Determine if the calculation type is Data Erasure: This function determines whether the current data processing request involves Data Erasure. Data Erasure is a data protection technology that splits the original data into multiple blocks and calculates additional check blocks. Even if some blocks are lost or damaged, the original data can be restored using the remaining intact blocks and check blocks.

[0045] Calling the data erasure calculation unit: Once the calculation type is confirmed to be data erasure, the storage client calls the preset data erasure calculation unit through the first processing unit.

[0046] Performing encoding calculations: After receiving the first data stream from the storage client, the data erasure calculation unit segments the first data stream and performs encoding calculations on the segmented data blocks based on the erasure coding algorithm. This process generates first erasure correction data and first check data. The first erasure correction data contains the segmented data blocks, while the first check data is an additional check block generated for data recovery.

[0047] Generating a data stream after coding calculation: After completing the coding calculation, the data erasure calculation unit combines the first erasure data and the first check data to generate a first data stream after coding calculation.

[0048] (2) When it is determined that the first calculation type is data encryption, the data encryption calculation unit in the first processing unit is called by the first processing unit, and the encryption algorithm and the first key corresponding to the first data stream are determined according to the data encryption calculation unit; the encryption calculation unit encrypts the first data stream according to the encryption algorithm and the first key to generate the first data stream after the encoding calculation.

[0049] It is understandable that, when the calculation type is determined to be data encryption, data encryption processing can be implemented by the first processing unit. The specific steps are:

[0050] Determine if the calculation type is data encryption: This function determines whether the calculation type of the current data operation is data encryption. Data encryption is an important means of protecting data security. It converts raw data into ciphertext to prevent unauthorized access during transmission or storage.

[0051] Invoking the Data Encryption Calculation Unit: If the current operation determines that data encryption is required, the storage client invokes the built-in Data Encryption Calculation Unit through the first processing unit. The Data Encryption Calculation Unit is a hardware-level unit specifically designed to execute data encryption algorithms. It can complete complex data encryption tasks in a very short time, improving data security and processing efficiency.

[0052] Determine the encryption algorithm and key: After receiving the first data stream, the data encryption calculation unit will determine the applicable encryption algorithm and first key according to preset or client-specified conditions.

[0053] Performing encryption: The data encryption calculation unit encrypts the first data stream using a predetermined encryption algorithm and key. The encryption process converts the original data into ciphertext data that is not directly recognizable, generating the encoded first data stream.

[0054] Optionally, after storing the first data stream after encoding calculation in the second address space of the cache unit in the first processing unit in the above step S202, the method further includes: when the first processing unit determines that the first data stream after encoding calculation has been stored in the second address space, determining the first identifier corresponding to the second address space; and sending the completion status and the first identifier corresponding to the first data stream after encoding calculation to the storage client through the write preparation interface corresponding to the first processing unit, wherein the completion status is used to indicate that the first data stream after encoding calculation has been stored in the second address space.

[0055] It is understandable that after the data processing is completed, the storage status and key identification information can be fed back to the storage client through the first processing unit. Specifically:

[0056] Confirming data storage: After the first processing unit completes the encoding calculation (such as erasure coding or encryption operation) of the first data stream, and the first data stream after the encoding calculation has been successfully stored in the second address space in the cache unit, the first processing unit will confirm that the storage status of the first data stream after the encoding calculation is successful storage.

[0057] Determining an identifier: After confirming successful data storage, the first processing unit determines a first identifier associated with the storage location (second address). The first identifier is key information used to uniquely identify the storage location of the first data stream after the encoded calculation in the cache unit of the first processing unit.

[0058] Sending completion status and first identifier: The first processing unit sends the data processing completion status and the first identifier to the storage client via a predefined write preparation interface. The completion status is typically a Boolean value or status code that explicitly indicates whether the encoded first data stream has been successfully stored in the second address space.

[0059] The client receives feedback: After receiving the completion status and first identifier from the first processing unit, the storage system client determines whether the data storage was successful based on the completion status. If the storage is successful, the storage client can use the first identifier to perform subsequent data operations, such as writing, reading, and retrieving data. If the storage fails, the storage client needs to retry or perform exception handling based on the error information.

[0060] Optionally, the above-mentioned step S204 constructs a write control message based on the first address of the first solid-state hard disk corresponding to the first data stream and the first identifier of the second address corresponding to the second address space, including: obtaining the first data information of the first data stream after the encoding calculation, and determining the message format of the write control message to be constructed, wherein the message format includes: the header information and one or more data segments of the write control message, the first data information includes at least one of the following: the data stream size of the first data stream after the encoding calculation, the creation time of the first data stream after the encoding calculation, the data version corresponding to the first data stream after the encoding calculation, and the data write instruction corresponding to the first data stream after the encoding calculation; filling the header information according to the first address and the first identifier, and filling the one or more data segments according to the first data information, so as to construct the write control message according to the filled header information and the filled one or more data segments.

[0061] It is understandable that the process of constructing a write control message may include:

[0062] Obtaining First Data Information: After the first processing unit completes encoding calculations (such as erasure correction and encryption) for the first data stream, it obtains first data information about the first data stream after the encoding calculations. The first data information may include the size of the data stream, creation time, data version, or data write instructions.

[0063] Determine the message format: Determine the message format of the write control message to be constructed. A write control message typically consists of a header and one or more data segments. The header contains metadata required for control and data location, while the data segment contains the actual data information or instructions.

[0064] Filling header information: When constructing the message, the header information will be filled according to the first address (i.e., the target address of the storage device) and the first identifier (such as the key value, which is used to uniquely identify the storage location).

[0065] Filling the data segment: The data segment is filled based on the first data information. The information in the data segment describes the characteristics of the data stream and storage requirements. For example, the data size helps determine storage space, the creation time or data version is used for data version control and consistency management, and the data write instruction directly guides the specific execution of the storage operation.

[0066] Constructing a write control message: A complete write control message is constructed based on the filled header information and data segment. Once the write control message is constructed, it will be sent to the first storage node to notify the first storage node to store the data.

[0067] Optionally, the above-mentioned data writing method may include not only a data writing process, but also a data reading process, specifically: when a data reading request is received, the second data information of the third data stream corresponding to the data reading request is obtained through the first processing unit, wherein the second data information includes at least one of the following: a third address of the second solid-state hard disk storing the third data stream, and calculation parameters corresponding to the third data stream, and the calculation parameters include at least one of the following: data correction and erasure parameters, data encryption parameters, and data deduplication parameters; the first processing unit allocates a third address space to the third data stream in the cache unit, and sends the second identifier of the fourth address corresponding to the third address space to the storage client; and performs the data reading operation corresponding to the data reading request according to the third address and the second identifier.

[0068] Wherein, the data reading operation corresponding to the data reading request is executed according to the third address and the second identifier, including: constructing a read control message according to the third address and the second identifier, and sending the read control message to the second storage node corresponding to the second solid-state hard disk, so that the second storage node obtains the third data stream in the second solid-state hard disk according to the read control message, and sends the third data stream to the third address space; controlling the first processing unit to perform a second calculation type of decoding calculation on the third data stream stored in the third address space according to the calculation parameters, and updating the data in the third address space to the third data stream after decoding calculation, so that the target object sending the data reading request reads the third data stream after decoding calculation, wherein the second calculation type includes at least one of the following: data erasure, data encryption, and the second calculation type is the calculation type of the third data stream.

[0069] It is understandable that the storage client can also process data read requests, specifically:

[0070] Receiving a data read request: When the storage client receives a data read request from the target object, the first processing unit processes the data read request. The data read request typically includes information such as the third data stream, the read position (offset), and the length.

[0071] Obtaining read data information: The first processing unit analyzes the data read request and extracts the second data information, including the third address of the second solid-state drive storing the third data stream (i.e., the data storage address in the SSD) and calculation parameters corresponding to the third data stream. The calculation parameters can be data erasure parameters, data encryption parameters, or data deduplication parameters, depending on the type of encoding calculation performed when the data was written.

[0072] Allocating a cache address space: The first processing unit allocates a third address space in the cache unit for the third data stream. The third address space is used to temporarily store data read from the second solid-state drive and subsequent decoding calculation results. Simultaneously, the second identifier of the fourth address corresponding to the allocated third address space is sent to the storage client.

[0073] Constructing and sending a read control message: The storage client constructs a read control message based on the received second identifier and third address, and sends the message to the second storage node where the second solid-state drive storing the third data stream is located. The read control message includes information such as a read instruction, a key value, and a target device address (i.e., the third address). It instructs the second storage node to read data from the second solid-state drive and transfer it to the third address space.

[0074] Execute data reading: After the second storage node receives the read control message, it reads the third data stream from the third address of the second solid-state drive according to the instructions and information contained in the read control message, and directly writes the data to the third address space through the second processing unit in the second storage node, avoiding additional data movement between the CPU and memory.

[0075] Decoding calculation: After the third data stream is read into the third address space, the first processing unit performs a decoding calculation of the second calculation type on the third data stream based on the calculation parameters corresponding to the data read request. If the data was erasure coded when written, erasure decoding calculation is performed at this time; if the data was encrypted when written, data decryption calculation is performed at this time. The decoded third data stream is then updated to the third address space.

[0076] Completion of the read operation: After the decoding calculation is completed, the decoded data stored in the third address space can be directly read by the target object.

[0077] Optionally, before obtaining the second data information of the third data stream corresponding to the data read request through the first processing unit, the method also includes: obtaining a data stream identifier corresponding to the third data stream according to the data read request, and determining distribution information corresponding to the third data stream according to the data stream identifier, wherein the distribution information is used to indicate the distribution of the third data stream in multiple storage nodes; parsing the distribution information, and determining the second storage node storing the third data stream among the multiple storage nodes based on the parsed distribution information; and determining the address corresponding to the second solid-state hard disk in the second storage node as the third address.

[0078] It is understandable that this description reveals that in the technical solution of the present invention, the storage client needs to determine the specific location of the third data stream to be read in the solid-state drive (i.e., the third location). Specifically:

[0079] Obtaining a data stream identifier: When the storage client receives a data read request, it extracts the data stream identifier corresponding to the third data stream from the data read request. A data stream identifier is typically a unique identifier for a data stream in the storage system, such as a file hash value or an object ID, and is used to accurately identify and locate data.

[0080] Determining data distribution information: Using the data stream identifier, the storage client queries the storage system's metadata to determine the distribution information for the third data stream. This distribution information describes the storage layout of the third data stream across multiple storage nodes, including the storage node where each striped data block is stored and the specific storage location within the storage node (e.g., address within an SSD).

[0081] Parsing the distribution information: The storage client parses the distribution information to identify the second storage nodes storing the third data stream and the storage locations within the second storage nodes. The parsing process may involve converting the distribution information into a format that is easier to process, such as mapping the node ID and address in the distribution information to actual network addresses and storage device addresses.

[0082] Determining the third address: From the parsed distribution information, the storage client can determine the second storage node storing the third data stream. The second storage node is the node directly accessed for data read requests. It contains the third address required to read the data stream, specifically the specific location of the third data stream on the second SSD. This specific location is the third address.

[0083] Optionally, obtaining the second data information of the third data stream corresponding to the data read request through the first processing unit includes: calling the read preparation interface corresponding to the first processing unit, and sending the second data information to the first processing unit according to the read preparation interface.

[0084] It is understandable that the data interaction process between the storage client and the first processing unit can be implemented through the read preparation interface.

[0085] Optionally, after updating the data in the third address space to the third data stream after decoding and calculation, the method further includes: determining the application cache address corresponding to the data read request, wherein the application cache address is the address corresponding to the application cache space in the target object for reading the third data stream after decoding and calculation; calling the read interface corresponding to the first processing unit, and sending the second identifier and the application cache address to the first processing unit according to the read interface; upon receiving the second identifier and the application cache address, the first processing unit sends the third data stream after decoding and calculation to the application cache space corresponding to the application cache address, so that the target object reads the third data stream after decoding and calculation.

[0086] It is understandable that the application cache address can be determined, and the first processing unit can be used to quickly transmit the decoded and calculated third data stream to the address space corresponding to the application cache address after data decoding. The specific steps include:

[0087] Determine the application cache address: When a storage client receives a data read request, it determines the application cache address within the target object to which the data should be read. The application cache address is the specific address of the application cache space within the target object used to read and process the decoded third data stream.

[0088] Calling the read interface: The storage client can call the read interface corresponding to the first processing unit. The read interface is a hardware interface used to control data reading and transmission. The storage client sends the second identifier (i.e., the key value in the third address space) and the application cache address to the first processing unit through the read interface.

[0089] Sending the second identifier and application cache address: The storage client sends the second identifier and application cache address to the first processing unit via the read interface. The second identifier is used to locate the decoded and calculated third data stream in the cache unit of the first processing unit, and the application cache address is used to indicate the application cache space within the target object.

[0090] Data transfer to the application cache: Upon receiving the second identifier and the application cache address, the first processing unit immediately initiates the data transfer process. The decoded and calculated third data stream is written directly from the address space corresponding to the third address to the cache space corresponding to the target object's application cache address via DMA (Direct Memory Access). DMA technology allows data to be transferred directly between hardware without CPU intervention, significantly improving data transfer speeds and reducing CPU and memory usage.

[0091] The target object reads data: Once the data transmission is completed, the target object can directly read the decoded and calculated third data stream from the application cache space.

[0092] Optionally, the first processing unit performs a decoding calculation of a second calculation type on the third data stream stored in the third address space according to the calculation parameters, including: when the second calculation type is data erasure correction, calling the data erasure correction calculation unit in the first processing unit through the first processing unit, so that the data erasure correction calculation unit determines the second erasure correction data and second verification data corresponding to the third data stream according to the calculation parameters; and performing a decoding calculation on the third data stream according to the second erasure correction data and the second verification data.

[0093] It is understandable that the first processing unit may perform data erasure decoding calculation on the third data stream, specifically:

[0094] Invoking the Data Erasure Correction Calculation Unit: If the first processing unit determines that the second computation type corresponding to the data read request is data erasure decoding, the first processing unit invokes its own built-in Data Erasure Correction Calculation Unit. The Data Erasure Correction Calculation Unit is part of the hardware acceleration module and is specifically designed to perform erasure code decoding calculations to restore the original data.

[0095] Determining erasure data and check data: Upon receiving the call, the erasure calculation unit determines the second erasure data and second check data in the third data stream based on pre-provided calculation parameters (including but not limited to the erasure algorithm, data block size, and redundancy ratio). Erasure data refers to data blocks that are stored in a distributed manner after erasure coding, while check data refers to additional data blocks generated for data redundancy and recovery.

[0096] Perform decoding calculation: After determining the second erasure correction data and the second check data, the data erasure correction calculation unit performs decoding calculation on the third data stream to restore the original third data stream and obtain the third data stream after decoding calculation.

[0097] Updating decoded data: After the decoding calculation is completed, the third data stream stored in the cache unit in the first processing unit will be updated to the decoded third data stream.

[0098] Optionally, the first processing unit performs a decoding calculation of a second calculation type on the third data stream stored in the third address space according to the calculation parameters, including: when the second calculation type is data encryption, calling the data encryption calculation unit in the first processing unit through the first processing unit, and obtaining the second key and second encryption algorithm when encrypting the third data stream according to the data encryption calculation unit; determining the decryption algorithm corresponding to the second encryption algorithm, and decrypting the third data stream according to the decryption algorithm and the second key to generate the third data stream after the decoding calculation.

[0099] It is understandable that the first processing unit may further perform a decoding operation corresponding to the data encryption, that is, decrypt the third data stream. Specifically:

[0100] Calling the data encryption calculation unit: After determining that the second calculation type of the data read request is data decryption, the first processing unit calls its built-in data encryption calculation unit.

[0101] Obtaining encryption information: Through the data encryption calculation unit, the first processing unit can obtain the second key and the second encryption algorithm used when encrypting the third data stream.

[0102] Determine the decryption algorithm: The first processing unit determines the corresponding decryption algorithm based on the obtained second encryption algorithm. The encryption algorithm and the decryption algorithm are one-to-one.

[0103] Decryption: The first processing unit decrypts the encrypted third data stream retrieved from the storage node using the specified decryption algorithm and the second key, restoring the data to its plaintext state. The decryption process may include key expansion, data block decoding, and algorithm-specific reverse operations.

[0104] Generating a decoded third data stream: After decryption, the originally encrypted third data stream is restored to an unencrypted state, generating a decoded third data stream. The decoded third data stream is stored in the cache unit of the first processing unit.

[0105] In order to better understand the process of the above-mentioned data writing method, the implementation process of the above-mentioned data writing method is described below in combination with optional embodiments, but it is not used to limit the technical solution of the embodiments of this application.

[0106] The following are explanations of the technical terms that may appear in the optional embodiments of this application:

[0107] 1) Single-stream bandwidth: the read and write bandwidth of a single application thread;

[0108] 2) High performance computing (HPC): refers to a computing model that uses supercomputers and parallel processing technology to solve large-scale complex computing problems in science, engineering, business and other fields;

[0109] 3) Data striping: Split a file into many small data blocks, i.e. data striping, and then distribute the data strips to each node of the distributed storage;

[0110] 4) Kernel state: The kernel of a computing processor is in operation. Modern operating systems such as Linux and Windows have kernel state, which is used to run operating system management processes, resource scheduling, memory management, and other processes.

[0111] 5) User state: The user running state of the computing processor. Modern operating systems such as Linux and Windows have user state, which is used to run user processes.

[0112] 6) Data Copy: Modern computer processors have memory addresses divided into user state and kernel state. User state is the address space of user processes, which is independently allocated to each user process. Kernel state's address space is used by the operating system kernel process and cannot be used by user state.

[0113] 7) Computer Operating System (OS): It is a computer program that manages computer hardware and software resources, controls the operation of the computer, manages all system resources, and provides users with a user-friendly operating interface to perform various tasks.

[0114] 8) Kernel client, which refers to the distributed storage client, is deployed in the user state of the client host OS to enable interconnected access between the client host and the distributed storage system;

[0115] 9) Virtual file system (VFS) interface: VFS implements the file interface of the operating system, and various file systems implement statistical interface docking;

[0116] 10) Memory Management Unit (MMU): Responsible for the virtual address space management of each process in the operating system, and responsible for mapping and managing the address space of user-mode processes and physical memory.

[0117] The optional embodiments of the present application relate to the field of storage systems, specifically subdivided into the field of independent client performance optimization. A data stream acceleration method for a storage system is proposed, and it is proposed to use computational erasure correction technology in the process of network data stream processing, merge data stream processing and hardware acceleration, and improve the processing performance of a single stream.

[0118] Single-stream storage is a demanding application scenario in the storage field, primarily suited for high-performance computing applications, especially data writing applications such as satellites, astronomical telescopes, and cryo-electron microscopes. As data volumes from satellites and astronomical telescopes increase, distributed storage clients are required to provide higher reception performance, specifically single-stream performance.

[0119] Distributed storage is generally defined as a storage system composed of multiple storage nodes interconnected via a network, creating a unified namespace that can be accessed concurrently by clients. Distributed storage achieves single-threaded performance by striping files and sending them concurrently to multiple nodes for parallel processing.

[0120] In order to solve the problem of memory resource preemption caused by the CPU running software to execute encoding calculation tasks in data stream processing, the CPU also needs to handle other business while processing encoding calculations. A method for implementing data stream processing through a Peripheral Component Interconnect Express (PCIE) expansion card has been proposed in related technologies. Specifically, the PCIE expansion card is directly connected to the computing node, and data processing is completed by transmitting the data to the accelerator card before returning it to the original processing flow. The advantage of this processing is that the accelerator card and the original processing flow are decoupled and plug-and-play. The problem is that the interactive resource consumption is large, which limits the processing performance of the accelerator card.

[0121] Figure 3 1 is a flow chart of a method for processing data streams using a peripheral component interconnect express (PCIE) expansion card in the related art. Figure 3 As shown:

[0122] Step S301: The application process in user mode sends a request to read or write data;

[0123] Step S302: After receiving the processed read / write data request sent by the kernel-mode VFS interface, perform other processing operations on the file corresponding to the processed read / write request, such as pre-processing operations.

[0124] Step S303: copy and migrate the data to the PCIE expansion card;

[0125] Step S304: perform erasure redundancy calculation on the data through the PCIE expansion card, and send the data obtained after iusa years to the accelerator card;

[0126] Step S305 : Send the data processed by the accelerator card to the backend storage system, that is, any one or more storage nodes among the storage nodes 31 - 36 .

[0127] An optional embodiment of the present application proposes a method for accelerating data streams in a storage system. By coordinating data stream processing and erasure calculations on a distributed storage system client, the main data stream processing and erasure calculation sub-processing are integrated, avoiding data migration and interaction from the main data to the erasure sub-processing stream, and achieving hardware acceleration within the data stream. Specifically:

[0128] The data flow acceleration method for storage systems involves three parts: the storage system client (i.e., storage client), the data processing unit (DPU) / SmartNIC (i.e., first processing unit and second processing unit), and the storage system node (i.e., target storage node). These three parts can execute the data writing and reading processes. Specifically:

[0129] Figure 4 This is an architectural diagram corresponding to a write process flow according to an optional embodiment of the present application. Specifically, the write process flow may include the following process steps:

[0130] (1) The storage system client completes the metadata processing, data striping and addressing operations of the file system client, negotiates consistency of the write address space, interacts with the DPU / Smart NIC for data stream processing, and passes the write data stream to the DPU / Smart NIC for calculation through the hardware acceleration interface;

[0131] (2) The client's DPU / Smart NIC contains an FPGA that can perform accelerated calculations for erasure correction / encryption of the write data stream (erasure correction calculations can generate erasure codes). After the calculation is completed, it only returns the address of the cache unit in the client's DPU / Smart NIC where the data calculated by the client is stored (i.e. Figure 4The DPU address in the ), does not return the data stream (where, Figure 4 Key: addr is the address identifier corresponding to the address in the DPU);

[0132] An optional embodiment of the present application can implement data communication between the DPU and storage nodes, other DPUs or network devices through a communication module.

[0133] (3) The storage client and the storage node complete the coordination of the write control message of the data flow, that is, the storage client sends a write control message to the storage node, so that the storage node pulls the data in the cache unit in the DPU smart network card through the hardware acceleration interface based on the write control message.

[0134] (4) Through the DPU smart network card and storage medium device (SSD) in the storage node, the data is written using Direct Memory Access (DMA).

[0135] Among them, the hardware acceleration interfaces involved in the data writing process may include: preprocessing interface: transmitting data, erasure calculation parameters, returning the DPU key and cache address; pulling interface: storage node DPU / intelligent pulling data, etc.

[0136] Among them, NIC is used as a communication bridge between storage nodes and DPU / Smart NIC when DPU and Smart NIC are used in combination, receiving and sending data streams.

[0137] In an alternative embodiment, Figure 5 is a flowchart of writing data processing according to an optional embodiment of the present application, such as Figure 5 As shown:

[0138] Step S501: The user-mode application writes the specified file, offset, and length through system calls and VFS, and transfers the data to the storage system client cache.

[0139] In step S502 , the storage system client (ie, storage client) completes metadata and data processing and finds a target node and a target device according to a distributed data layout algorithm.

[0140] Step S503: Before sending the write request, the storage system client determines whether there is a reserved destination address space. If so, the address space is used directly. If there is no reserved address space, the client performs consistency negotiation with the target node (i.e., the first storage node) and the target device to allocate the destination device address space (i.e., the first address space).

[0141] In step S504, the storage system client calls the hardware acceleration interface (write preparation interface) of the DPU / SmartNIC (i.e., the first processing unit) and passes the cached data (i.e., the first data stream), the address length of the cached data, and the calculation type (i.e., the first calculation type) of the cached data to the DPU / SmartNIC. The calculation type may include erasure correction, encryption, deduplication, etc.

[0142] In step S505, the DPU / SmartNIC performs erasure / encryption encoding calculations on the incoming data cache based on the calculation type (including erasure correction and encryption) and the data information. (If erasure correction calculation is performed, the erasure processing interface is called to send the data-related information to the erasure calculation module in the DPU / SmartNIC; if encryption calculation is performed, the data-related information is sent to the encryption calculation module.) The processed data is written to the data cache (i.e., the cache unit) in the DPU / SmartNIC.

[0143] The erasure calculation module is a data erasure calculation unit, and the encryption calculation module is a data encryption calculation unit.

[0144] Furthermore, the DPU / smart network card returns the result to the storage system client through the hardware acceleration interface-write preparation interface. The interface returned by the interface includes: the completion status of the interface, and the key value (i.e., the first identifier) of the address of the processed data in the data cache. The key value is the key representing the address of the processed data cached by the cache unit in the DPU / smart network card.

[0145] The network module is responsible for sending and receiving data packets, for example, receiving cache data sent by the storage system client, the address length of the cache data, and the calculation type of the cache data.

[0146] The storage system client can also pre-apply for write space in the DPU's data cache to store processed data.

[0147] Step S506: The storage client sends the calculated data cache key (i.e., the first identifier) to the storage node in the form of a write control message according to the address of the target node and the target device (i.e., the first address). Figure 5 One or more of the storage nodes 51 to 53 in the storage node may be the first storage node);

[0148] In step S507, the storage node receives the write control message sent by the storage client, calls the DPU / smart network card hardware acceleration interface - write pull interface, and passes the key to the DPU / smart network card of the storage node; then the DPU / smart network card of the storage node pulls the data according to the key and the second address; the DPU / smart network card of the storage client receives the data pull request, and transmits the data in the data cache (that is, the first data stream after encoding calculation) to the DPU / smart network card of the storage node; the DPU / smart network card of the storage node writes the first data stream after encoding calculation into the corresponding address space in the SSD medium.

[0149] The storage node responds successfully to the write control message. After receiving all responses, the storage system client completes the write request and returns the application's system call.

[0150] Figure 6 : This is an architectural diagram corresponding to a read processing flow according to an optional embodiment of the present application. Specifically, the read processing flow may include the following process steps:

[0151] (1) The storage system client completes the metadata processing, data striping and addressing operations of the file system client. The data stream processing pre-applies for the cache address and key (second identifier) of the DPU / Smart NIC through the hardware acceleration interface;

[0152] (2) The storage system client sends the pre-requested read cache address and key, the target device and the address of the target node to read the data to the storage node through the read control message of the data stream;

[0153] (3) The storage node transmits the target address of the data, the address of the read cache, and the key to the DPU / Smart NIC of the storage node;

[0154] (4) The DPU / Smart NIC triggers the read operation of the storage medium / device, and reads the data from the storage medium device DMA to the cache space of the DPU / Smart NIC;

[0155] (5) The DPU / Smart NIC of the storage node pushes the data to the DPU / Smart NIC cache of the client, completing the reading process of the data stream.

[0156] In an alternative embodiment, Figure 7 is a flowchart of the data reading process according to an optional embodiment of the present application, such as Figure 7 As shown:

[0157] Step S701: The application reads the specified file, offset, and length through a system call and VFS, and passes in the read cache address.

[0158] Step S702: The storage client processes a read request (i.e., a data read request);

[0159] Step S703: Find the address of the target node and target device of the data (i.e., the third address) based on the file information.

[0160] Step S704: Call the hardware acceleration interface - read preparation interface to send the data length, erasure correction, encryption and other parameters (i.e., calculation parameters) of the data to be read to the DPU / intelligent network card (i.e., the first processing unit);

[0161] DPU / Smart NIC, based on the incoming data length, erasure, encryption and other parameters, calls the corresponding erasure processing interface or encryption processing interface, allocates internal cache and computing units (i.e. Figure 7 The erasure calculation module and encryption calculation module in the data) perform erasure / encryption decoding operations on the data;

[0162] The DPU / Smart NIC returns the key value (i.e., the second identifier) corresponding to the data cache in the DPU / Smart NIC through the hardware acceleration interface-read preparation interface.

[0163] Step S705: After receiving the return value of the Key value, the storage client sends the Key to the target node and target device (i.e. Figure 7 Storage nodes 71-73 in the storage node, that is, the second storage node).

[0164] In step S706, the storage system node receives the read control message and calls the DPU / SmartNIC hardware acceleration interface - the read interface, passing the key, client address, and target storage device address to the DPU / SmartNIC (i.e., the second processing unit) in the storage node. The DPU / SmartNIC in the storage node pushes the read data to the client's DPU / SmartNIC.

[0165] In step S707, the client's DPU / Smart NIC receives the pushed data and puts it into the cache of the specified key. After receiving all the data, it performs erasure correction / encryption decoding calculation and stores the decoded data in the data cache.

[0166] After the DPU / Smart NIC of the storage system node completes the data push, it returns a successful read control message to the storage client. The client successfully receives the read control message, calls the hardware acceleration interface - the read interface, and sends the key and application cache address to the client's DPU / Smart NIC. The client's DPU / Smart NIC copies the calculated cache data (that is, the decoded third data stream) to the application cache space based on the key and application cache address. The client returns the application's system call to complete the data read operation.

[0167] In summary, the optional embodiment of the present application proposes a hardware erasure acceleration technology based on data stream processing. Through the collaborative processing of data stream processing and erasure calculation on the distributed storage system client, the main data stream processing and erasure / encryption calculation sub-processing are integrated, avoiding the data migration and interaction from the main data to the erasure sub-processing stream, and realizing hardware acceleration within the data stream; on high-performance data streams, traditional hardware acceleration cards have the problem that the consumption of PICe data migration back and forth between the CPU and the acceleration card is far greater than the improvement of hardware acceleration, and the data interaction bottleneck problem causes the hardware acceleration to fail to achieve the expected performance improvement. The optional embodiment of the present application does not change the direction of data flow, accelerates the data stream, and improves the performance of data processing.

[0168] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0169] In this embodiment, a data writing system is also provided, which is used to implement the above embodiments and preferred implementations. Details that have already been described will not be repeated. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0170] Figure 8 is a framework diagram of a data writing system according to an embodiment of the present application, such as Figure 8 As shown, the system includes: a storage client 82 and a target storage node 84, wherein:

[0171] The storage client 82 is configured to, upon receiving a data write request, perform a decoding calculation of a first calculation type on a first data stream corresponding to the data write request through a first processing unit in the storage client, and store the decoded first data stream in a second address space of a cache unit in the first processing unit; construct a write control message based on a first address of a first solid-state drive corresponding to the first data stream and a first identifier of a second address corresponding to the second address space, and send the write control message to a first storage node including the first solid-state drive, wherein the first calculation type includes at least one of the following: data erasure and data encryption, the first calculation type is a calculation type of the first data stream, and the first address is an address corresponding to a first address space in the first solid-state drive for writing the decoded first data stream;

[0172] The target storage node 84 includes: the first storage node, which is used to obtain the first data stream after decoding and calculation according to the write control message, and write the first data stream after decoding and calculation into the first solid-state hard disk.

[0173] Through the data writing system of the present application, the first data stream is encoded and calculated by the first processing unit in the storage client, and the first data stream after the encoding calculation is stored in the cache unit in the first processing unit; then, according to the first address of the first solid-state hard disk to which the first data stream is to be written and the first identifier of the second address of the cache unit where the first data stream is located, the first storage node containing the first solid-state hard disk can write the first data stream after the encoding calculation to the first solid-state hard disk. In other words, the embodiment of the present application performs the encoding calculation task in the data stream processing by the storage client instead of the CPU, thereby solving the problem of memory resource preemption caused by the CPU running software to execute the encoding operation task in the data stream processing in the related art. When the CPU processes the encoding operation, it also needs to process other services, thereby avoiding the memory resource preemption problem.

[0174] In an exemplary embodiment, the first storage node is further used to parse the write control message upon receiving the write control message to determine the first address and the first identifier; call the write pull interface corresponding to the second processing unit in the first storage node, and send the first address and the first identifier to the second processing unit according to the write pull interface; the second processing unit is also used to determine the second address corresponding to the first data stream after decoding and calculation according to the first identifier, and send a data pull request to the first processing unit according to the second address, so that the first processing unit sends the first data stream after decoding and calculation to the second processing unit according to the data pull request; and write the first data stream after decoding and calculation into the first address space corresponding to the first address.

[0175] In an exemplary embodiment, the storage client 82 is further configured to obtain a second data stream corresponding to the data write request and perform data preprocessing on the second data stream to obtain the first data stream; determine, based on a distributed data layout algorithm, whether a reserved address space corresponding to the first data stream exists in a plurality of solid-state drives, wherein each storage node includes a solid-state drive, and the plurality of storage nodes include: the first storage node, and the reserved address space is an address space reserved for writing the first data stream after the encoding calculation; if it is determined that the reserved address space exists in one or more second solid-state drives among the plurality of solid-state drives, determine the addresses corresponding to the one or more second solid-state drives as the first address, wherein the first solid-state drive includes: the one or more second solid-state drives; if it is determined that the reserved address space does not exist in the plurality of solid-state drives, perform consistency negotiation with the plurality of storage nodes so that the plurality of storage nodes select one or more third solid-state drives from the plurality of solid-state drives and allocate address space for the first data stream in the one or more third solid-state drives; determine the addresses corresponding to the one or more third solid-state drives as the first address, wherein the first solid-state drive includes: the one or more third solid-state drives.

[0176] In an exemplary embodiment, the storage client 82 is further used to, when determining that the first calculation type is data erasure, call the data erasure calculation unit in the first processing unit through the first processing unit, and encode the first data stream according to the data erasure calculation unit to generate first erasure data and first verification data corresponding to the first data stream; and generate the first data stream after the encoding calculation according to the first erasure data and the first verification data.

[0177] In an exemplary embodiment, the storage client 82 is also used to call the data encryption calculation unit in the first processing unit through the first processing unit when it is determined that the first calculation type is data encryption, and determine the encryption algorithm and first key corresponding to the first data stream according to the data encryption calculation unit; the encryption calculation unit encrypts the first data stream according to the encryption algorithm and the first key to generate the first data stream after the encoding calculation.

[0178] In an exemplary embodiment, when the first processing unit determines that the first data stream after the encoding calculation has been stored in the second address space, the first identifier corresponding to the second address space is determined; the completion status and the first identifier corresponding to the first data stream after the encoding calculation are sent to the storage client 82 through the write preparation interface corresponding to the first processing unit, wherein the completion status is used to indicate that the first data stream after the encoding calculation has been stored in the second address space.

[0179] In an exemplary embodiment, the storage client 82 is also used to obtain the first data information of the first data stream after the encoding calculation, and determine the message format of the write control message to be constructed, wherein the message format includes: the header information of the write control message and one or more data segments, the first data information includes at least one of the following: the data stream size of the first data stream after the encoding calculation, the creation time of the first data stream after the encoding calculation, the data version corresponding to the first data stream after the encoding calculation, and the data write instruction corresponding to the first data stream after the encoding calculation; fill in the header information according to the first address and the first identifier, and fill in the one or more data segments according to the first data information, so as to construct the write control message according to the filled header information and the filled one or more data segments.

[0180] In an exemplary embodiment, the storage client 82 is also used to determine the second data information of the third data stream corresponding to the data read request when receiving the data read request, wherein the second data information includes at least one of the following: the third address of the second solid-state hard disk storing the third data stream, and the calculation parameters corresponding to the third data stream, and the calculation parameters include at least one of the following: a second data erasure parameter, a second data encryption parameter, and a second data deduplication parameter; the first processing unit in the storage client 82 is also used to allocate a third address space for the third data stream in the cache unit in the first processing unit, and send the second identifier of the fourth address corresponding to the third address space to the storage client; the storage client 82 is also used to construct a read control message based on the third address and the second identifier, and send the read control message to the second storage node corresponding to the second solid-state hard disk, wherein the target storage node includes: the second storage node.

[0181] In an exemplary embodiment, the second storage node is used to obtain the third data stream in the second solid-state hard disk according to the read control message, and send the third data stream to the third address space; the first processing unit is also used to perform a second calculation type of decoding calculation on the third data stream stored in the third address space according to the calculation parameters, and update the data in the third address space to the decoded third data stream, so that the storage client reads the decoded third data stream according to the second identifier, wherein the second calculation type includes at least one of the following: data erasure, data encryption, and data deduplication, and the second calculation type is the calculation type of the third data stream.

[0182] In an exemplary embodiment, the storage client 82 is further configured to call a read preparation interface corresponding to the first processing unit, and send the second data information to the first processing unit according to the read preparation interface.

[0183] In an exemplary embodiment, the storage client 82 is also used to determine the application cache address corresponding to the data read request, wherein the application cache address is the address corresponding to the application cache space in the target object for reading the third data stream after decoding and calculation; calling the read interface corresponding to the first processing unit, and sending the second identifier and the application cache address to the first processing unit according to the read interface; upon receiving the second identifier and the application cache address, the first processing unit sends the third data stream after decoding and calculation to the application cache space corresponding to the application cache address, so that the target object reads the third data stream after decoding and calculation.

[0184] In an exemplary embodiment, the storage client 82 is also used to call the data erasure correction calculation unit in the first processing unit through the first processing unit when the second calculation type is data erasure correction, so that the data erasure correction calculation unit determines the second erasure correction data and second verification data corresponding to the third data stream according to the calculation parameters; and performs decoding calculation on the third data stream according to the second erasure correction data and the second verification data.

[0185] In an exemplary embodiment, the storage client 82 is also used to call the data encryption calculation unit in the first processing unit through the first processing unit when the second calculation type is data encryption, and obtain the second key and second encryption algorithm for encrypting the third data stream according to the data encryption calculation unit; determine the decryption algorithm corresponding to the second encryption algorithm, and decrypt the third data stream according to the decryption algorithm and the second key to generate the third data stream after the decoding calculation.

[0186] For the description of the features in the embodiment corresponding to the data writing system, reference can be made to the relevant description of the embodiment corresponding to the data writing method, which will not be repeated here.

[0187] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above data writing method embodiments.

[0188] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned data writing method embodiments when running.

[0189] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0190] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data writing method embodiments are implemented.

[0191] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned data writing method embodiments are implemented.

[0192] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] The above is a detailed introduction to a data writing method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data writing method, characterized in that: Applicable to storage clients, including: Upon receiving a data write request, performing, by a first processing unit in the storage client, an encoding calculation of a first calculation type on a first data stream corresponding to the data write request, and storing the encoded first data stream in a second address space of a cache unit in the first processing unit, wherein the first calculation type includes at least one of the following: data erasure and data encryption, and the first calculation type is a calculation type of the first data stream; Constructing a write control message according to a first address of the first solid-state drive corresponding to the first data stream and a first identifier of a second address corresponding to the second address space, wherein the first address is an address corresponding to a first address space in the first solid-state drive for writing the first data stream after the encoded calculation; Sending the write control message to a first storage node including the first solid-state drive, so that the first storage node obtains the first data stream after encoding and calculation according to the write control message, and writes the first data stream after encoding and calculation into the first solid-state drive; The method further comprises: Upon receiving a data read request, obtaining, by the first processing unit, second data information of a third data stream corresponding to the data read request, wherein the second data information includes at least one of the following: a third address of a second solid-state drive storing the third data stream, and calculation parameters corresponding to the third data stream, the calculation parameters including at least one of the following: a data erasure parameter, a data encryption parameter, and a data deduplication parameter; allocating a third address space for the third data stream in the cache unit by the first processing unit, and sending a second identifier of a fourth address corresponding to the third address space to the storage client; A read control message is constructed according to the third address and the second identifier, and the read control message is sent to a second storage node corresponding to the second solid-state drive, wherein the target storage node includes: the second storage node.

2. The data writing method according to claim 1, wherein: Before constructing a write control message according to the first address of the first solid-state hard disk corresponding to the first data stream and the first identifier of the second address corresponding to the second address space, the method further includes: Acquire a second data stream corresponding to the data write request, and perform a data preprocessing operation on the second data stream to obtain the first data stream; Determining, according to a distributed data layout algorithm, whether a reserved address space corresponding to the first data stream exists in a plurality of solid-state drives, wherein each storage node includes a solid-state drive, and the plurality of storage nodes include: the first storage node, and the reserved address space is an address space reserved for writing the first data stream after the encoding calculation; In a case where it is determined that the reserved address space exists in one or more second solid-state hard disks among the plurality of solid-state hard disks, determining addresses corresponding to the one or more second solid-state hard disks as the first address, wherein the first solid-state hard disk includes: the one or more second solid-state hard disks; When it is determined that the reserved address space does not exist in the plurality of solid-state drives, performing consistency negotiation with the plurality of storage nodes so that the plurality of storage nodes select one or more third solid-state drives from the plurality of solid-state drives, and allocate address space for the first data stream in the one or more third solid-state drives; Addresses corresponding to the one or more third solid-state hard drives are determined as the first address, wherein the first solid-state hard drive includes: the one or more third solid-state hard drives.

3. The data writing method according to claim 1, wherein: Performing, by a first processing unit in the storage client, an encoding calculation of a first calculation type on a first data stream corresponding to the data write request, including: When it is determined that the first calculation type is data erasure correction, calling, by the first processing unit, a data erasure correction calculation unit in the first processing unit, and encoding the first data stream according to the data erasure correction calculation unit to generate first erasure correction data and first check data corresponding to the first data stream; The first data stream after the encoding calculation is generated according to the first erasure correction data and the first check data.

4. The data writing method according to claim 1, wherein: Performing, by a first processing unit in the storage client, an encoding calculation of a first calculation type on a first data stream corresponding to the data write request, including: When it is determined that the first calculation type is data encryption, calling, by the first processing unit, a data encryption calculation unit in the first processing unit, and determining, according to the data encryption calculation unit, an encryption algorithm and a first key corresponding to the first data stream; The encryption calculation unit encrypts the first data stream according to the encryption algorithm and the first key to generate the first data stream after the encoding calculation.

5. The data writing method according to claim 1, wherein: After storing the first data stream after the encoding calculation in the second address space of the cache unit in the first processing unit, the method further includes: When the first processing unit determines that the first data stream after the encoding calculation has been stored in the second address space, determining the first identifier corresponding to the second address space; The completion status corresponding to the first data stream after the encoding calculation and the first identifier are sent to the storage client through the write preparation interface corresponding to the first processing unit, wherein the completion status is used to indicate that the first data stream after the encoding calculation has been stored in the second address space.

6. The data writing method according to claim 1, wherein: Constructing a write control message according to a first address of a first solid-state hard disk corresponding to the first data stream and a first identifier of a second address corresponding to the second address space, including: Obtaining first data information of the first data stream after the encoding calculation, and determining a message format of a write control message to be constructed, wherein the message format includes: header information and one or more data segments of the write control message, and the first data information includes at least one of the following: a data stream size of the first data stream after the encoding calculation, a creation time of the first data stream after the encoding calculation, a data version corresponding to the first data stream after the encoding calculation, and a data write instruction corresponding to the first data stream after the encoding calculation; The header information is filled according to the first address and the first identifier, and the one or more data segments are filled according to the first data information, so as to construct the write control message according to the filled header information and the filled one or more data segments.

7. The data writing method according to claim 1, wherein: Performing a data read operation corresponding to the data read request according to the third address and the second identifier includes: Constructing a read control message according to the third address and the second identifier, and sending the read control message to the second storage node corresponding to the second solid-state drive, so that the second storage node obtains the third data stream in the second solid-state drive according to the read control message and sends the third data stream to the third address space; Control the first processing unit to perform a decoding calculation of a second calculation type on the third data stream stored in the third address space according to the calculation parameters, and update the data in the third address space to the third data stream after decoding calculation, so that the target object sending the data read request reads the third data stream after decoding calculation, wherein the second calculation type includes at least one of the following: data erasure, data encryption, and the second calculation type is the calculation type of the third data stream.

8. The data writing method according to claim 7, wherein: Before obtaining, by the first processing unit, second data information of the third data stream corresponding to the data read request, the method further includes: Obtaining a data stream identifier corresponding to the third data stream according to the data read request, and determining distribution information corresponding to the third data stream according to the data stream identifier, wherein the distribution information is used to indicate distribution of the third data stream in multiple storage nodes; parsing the distribution information, and determining the second storage node storing the third data stream among the plurality of storage nodes according to the parsed distribution information; An address corresponding to the second solid-state hard disk in the second storage node is determined as the third address.

9. The data writing method according to claim 7, wherein: Acquiring, by the first processing unit, second data information of the third data stream corresponding to the data read request, includes: A read preparation interface corresponding to the first processing unit is called, and the second data information is sent to the first processing unit according to the read preparation interface.

10. The data writing method according to claim 9, wherein: After updating the data in the third address space to the third data stream after decoding and calculation, the method further includes: Determine an application cache address corresponding to the data read request, wherein the application cache address is an address corresponding to an application cache space in the target object for reading the third data stream after decoding and calculation; calling a read interface corresponding to the first processing unit, and sending the second identifier and the application cache address to the first processing unit according to the read interface; Upon receiving the second identifier and the application cache address, the first processing unit sends the decoded and calculated third data stream to the application cache space corresponding to the application cache address, so that the target object reads the decoded and calculated third data stream.

11. The data writing method according to claim 9, wherein: The first processing unit performs a decoding calculation of a second calculation type on the third data stream stored in the third address space according to the calculation parameter, including: When the second calculation type is data erasure correction, calling, by the first processing unit, a data erasure correction calculation unit in the first processing unit, so that the data erasure correction calculation unit determines, according to the calculation parameters, second erasure correction data and second check data corresponding to the third data stream; Decoding calculation is performed on the third data stream according to the second erasure correction data and the second check data.

12. The data writing method according to claim 9, wherein: The first processing unit performs a decoding calculation of a second calculation type on the third data stream stored in the third address space according to the calculation parameter, including: In a case where the second calculation type is data encryption, calling, by the first processing unit, a data encryption calculation unit in the first processing unit, and obtaining, according to the data encryption calculation unit, a second key and a second encryption algorithm for encrypting the third data stream; Determine a decryption algorithm corresponding to the second encryption algorithm, and decrypt the third data stream according to the decryption algorithm and the second key to generate the decoded third data stream.

13. A data writing system, characterized in that: It includes storage clients and target storage nodes, where: The storage client is used to, upon receiving a data write request, perform a decoding calculation of a first calculation type on the first data stream corresponding to the data write request through the first processing unit in the storage client, and store the first data stream after decoding and calculation in the second address space of the cache unit in the first processing unit; construct a write control message according to the first address of the first solid-state hard disk corresponding to the first data stream and the first identifier of the second address corresponding to the second address space, and send the write control message to the first storage node including the first solid-state hard disk, wherein the first calculation type includes at least one of the following: data erasure, data encryption, the first calculation type is the calculation type of the first data stream, and the first address is the address corresponding to the first address space in the first solid-state hard disk for writing the first data stream after decoding and calculation; The target storage node includes: the first storage node, configured to obtain the first data stream after decoding and calculation according to the write control message, and write the first data stream after decoding and calculation into the first solid-state hard disk; In which, the storage client is also used to obtain second data information of the third data stream corresponding to the data read request through the first processing unit when receiving a data read request, wherein the second data information includes at least one of the following: a third address of the second solid-state hard disk storing the third data stream, and calculation parameters corresponding to the third data stream, and the calculation parameters include at least one of the following: data correction and erasure parameters, data encryption parameters, and data deduplication parameters; allocate a third address space for the third data stream in the cache unit through the first processing unit, and send a second identifier of the fourth address corresponding to the third address space to the storage client; and execute the data read operation corresponding to the data read request according to the third address and the second identifier.

14. The data writing system according to claim 13, wherein: include: The first storage node is further configured to, upon receiving the write control message, parse the write control message to determine the first address and the first identifier; calling a write-pull interface corresponding to the second processing unit in the first storage node, and sending the first address and the first identifier to the second processing unit according to the write-pull interface; The second processing unit is further configured to determine, based on the first identifier, the second address corresponding to the first data stream after decoding and calculation, and send a data pull request to the first processing unit based on the second address, so that the first processing unit sends the first data stream after decoding and calculation to the second processing unit according to the data pull request; The first data stream after decoding and calculation is written into the first address space corresponding to the first address.

15. The data writing system according to claim 13, wherein: include: The storage client is further used to construct a read control message based on the third address and the second identifier, and send the read control message to the second storage node corresponding to the second solid-state drive, wherein the target storage node includes: the second storage node.

16. The data writing system according to claim 15, characterized in that: include: The second storage node is configured to obtain the third data stream in the second solid-state drive according to the read control message, and send the third data stream to the third address space; The first processing unit is also used to perform a second calculation type of decoding calculation on the third data stream stored in the third address space according to the calculation parameters, and update the data in the third address space to the third data stream after decoding calculation, so that the storage client can read the third data stream after decoding calculation according to the second identifier, wherein the second calculation type includes at least one of the following: data correction and erasure, data encryption, and data deduplication, and the second calculation type is the calculation type of the third data stream.

17. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data writing method according to any one of claims 1 to 12 when executing the computer program.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data writing method according to any one of claims 1 to 12.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data writing method according to any one of claims 1 to 12 are implemented.

Images