Data storage methods, devices, systems, network processors and acceleration equipment

By introducing a network processor into the acceleration device to generate and store write-ahead operation logs, the problems of main processor computing resource consumption and latency are solved, and data access efficiency and performance are improved.

CN117742568BActive Publication Date: 2026-11-10HUAWEI TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202211115045.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-14
Publication Date
2026-11-10
Estimated Expiration
2042-09-14

AI Technical Summary

Technical Problem

During data access, the main processor of the acceleration device consumes computing resources, affecting performance, and the multiple interactions between the main processor and the network processor increase latency.

Method used

By introducing a network processor into the acceleration device, write-ahead operation logs are generated and saved to the log storage area of ​​the storage device, reducing the computational burden on the main processor. Write operation instructions are processed by the network processor, avoiding multiple interactions between the main processor and the network processor.

Benefits of technology

It saves the main processor computing resources of the acceleration device, improves the execution efficiency of write operation instructions, shortens latency, and ensures data consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117742568B_ABST
    Figure CN117742568B_ABST
Patent Text Reader

Abstract

This application discloses a data storage method, apparatus, system, network processor, and acceleration device, relating to the field of computer technology. The acceleration device of this application processes write operation instructions sent by the host through a network processor, which can save the computing resources of the acceleration device's main processor and improve the performance of the acceleration device. The network processor generates a write-ahead log based on the received write operation instructions and saves the write-ahead log to the log storage area in the storage device. These two steps are executed quickly, thus reducing the time the acceleration device takes to execute write operation instructions and improving the efficiency of the acceleration device in executing write operation instructions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data storage method, apparatus, system, network processor, and acceleration device. Background Technology

[0002] With the development of computer and network technologies, computing devices offer an increasing number of services, leading to higher demands on the computing power of the host computer. To conserve the host's computing power and improve its performance, acceleration devices can be configured to offload some data processing functions from the host to these devices. For example, in data storage systems, data access functions can be offloaded from the host to acceleration devices, allowing the host to access data from the storage device through the acceleration device.

[0003] During data access, all kinds of data access commands sent by the host to the acceleration device are executed by the main processor of the acceleration device, which consumes the computing resources of the main processor of the acceleration device and affects the performance of the acceleration device. Summary of the Invention

[0004] This application provides a data storage method, apparatus, system, network processor, and acceleration device, which can save the computing resources of the main processor of the acceleration device and improve the execution efficiency of write operation instructions.

[0005] In a first aspect, embodiments of this application provide a data storage method. This method can be applied to an acceleration device connected between a host and a storage device. The acceleration device may include an interface and a network processor. The method may include the following steps: the interface in the acceleration device receives a write operation instruction sent by the host and forwards the write operation instruction to the network processor in the acceleration device; the network processor generates a write-ahead log based on the write operation instruction and saves the write-ahead log to a log storage area in the storage device. The write-ahead log is used to save the data to be written carried in the write operation instruction to the data storage area in the storage device during playback via the storage controller of the storage device.

[0006] In the aforementioned data storage method, after the interface in the acceleration device receives the write operation command sent by the host, the network processor in the acceleration device generates a write-ahead log based on the received write operation command and saves the write-ahead log to the log storage area in the storage device. The above two steps are executed quickly. The acceleration device does not need to process the write operation command and write the data to be written to the data storage area of ​​the storage device, nor does it need to wait for the storage controller of the storage device to write the data to be written to the data storage area. Therefore, it can reduce the time for the acceleration device to execute the write operation command and improve the execution efficiency of the acceleration device for the write operation command. Furthermore, since the write operation command is processed by the network processor in the acceleration device, the main processor in the acceleration device does not need to process the write operation command, which can save the computing resources of the main processor in the acceleration device and improve the performance of the acceleration device.

[0007] In one possible implementation, the network processor saves the write-ahead operation log to the log storage area in the storage device. If the network processor determines that the write-ahead operation log has been saved, it sends an operation execution completion notification to the host through the interface.

[0008] In the above embodiments, the network processor saves the write-ahead operation log to the log storage area of ​​the storage device, and sends an operation execution completion notification to the host when it determines that the write-ahead operation log has been saved. It does not need to wait for the storage controller of the storage device to replay the write-ahead operation log, write the data to be written to the data storage area, and then send the operation execution completion notification to the host. Therefore, the latency of the host and the acceleration device executing write operation instructions can be shortened. Furthermore, this process does not require multiple interactions between the main processor of the acceleration device and the network processor to execute the write operation instructions, thereby further shortening the latency of executing write operation instructions.

[0009] In one possible implementation, if the interface receives a non-modification operation instruction from the host, it sends the instruction to the main processor in the acceleration device or the storage controller of the storage device for processing. The non-modification operation instruction includes any operation instruction other than a write operation instruction. For example, upon receiving a non-modification operation instruction, the main processor in the acceleration device or the storage controller of the storage device can perform a corresponding operation based on the specific type of the instruction. For instance, a non-modification operation instruction may include a read operation instruction. Upon receiving a read operation instruction, the main processor in the acceleration device or the storage controller of the storage device can read the corresponding data from the data storage area in the storage device based on the address information carried in the read operation instruction.

[0010] In the above embodiments, considering that non-modification operation instructions have low latency requirements and consume less computing resources of the processor, non-modification operation instructions can be sent to the main processor of the acceleration device for processing.

[0011] In one possible implementation, to prevent the storage controller from reading data that has not yet been written to the data storage area, the log storage area of ​​the storage device can be configured with at least one storage space, and each of the at least one storage space has a corresponding filter. When the network processor saves the write-ahead operation log to the log storage area in the storage device, it can save the write-ahead operation log to a first storage space in the log storage area, wherein the first storage space can be any of the at least one storage space. The write-ahead operation log includes the first address information carried in the aforementioned write operation instruction. After saving the write-ahead operation log to the first storage space in the log storage area, the network processor can update the first filter corresponding to the first storage space. The updated first filter is used to indicate that the write-ahead operation log corresponding to the first address information is stored in the first storage space, waiting to be replayed by the storage controller of the storage device.

[0012] When the network processor saves the write-ahead operation log to the first storage space in the log storage area, the first filter corresponding to the first storage space is stored in the acceleration device. The network processor receives a read operation command from the host and obtains the second address information carried in the read operation command. The read operation command instructs the retrieval of data from the data storage area based on the second address information.

[0013] The network processor can determine, based on the first filter, whether the write-ahead log corresponding to the second address information is stored in the first storage space. If it is determined that the write-ahead log corresponding to the second address information is stored in the first storage space, the network processor can send a delayed read notification to the storage controller of the storage device. The delayed read notification instructs the storage controller to wait for the write-ahead log corresponding to the second address information to be replayed before executing the aforementioned read operation instruction.

[0014] In the above implementation, by dividing the log storage area into one or more storage spaces and setting a filter for each storage space, the system determines whether the write-ahead log corresponding to the address information carried by the read operation instruction is located in a storage space and has not yet been replayed. If it has not been replayed, the execution of the read operation instruction is delayed, thereby avoiding the problem of the storage controller being unable to read data or reading incorrect data when executing the read operation instruction.

[0015] In one possible implementation, when the network processor saves the write-ahead operation log to a first storage space in the log storage area, the first filter corresponding to the first storage space is stored in the acceleration device. If the network processor determines that the first storage space is full, the first filter can be saved to the storage device to save storage space in the acceleration device.

[0016] In one possible implementation, if the network processor determines, based on the first filter, that the write-ahead log corresponding to the second address information is not stored in the first storage space, it sends a read operation instruction to the storage controller of the storage device. This allows the storage controller to determine, based on the filter in the storage device, whether the write-ahead log corresponding to the second address information is stored in the log storage area and awaits playback by the storage controller of the storage device. This can more effectively prevent the storage controller from reading data that has not yet been written to the data storage area.

[0017] In one possible implementation, the write-ahead log may include the numbers of the write operation instructions. The write operation instruction numbers may be sequentially incremented according to the time order in which the network processor receives each write operation instruction. When updating the first filter corresponding to the first storage space based on the first address information, at least one first target location can be determined in the first filter based on the first address information, and the numbers of the write operation instructions can be saved to each first target location respectively.

[0018] By storing the write operation instruction number in each first target location, compared with setting the element in each first target location to a set value in related technologies, the probability of misjudgment can be reduced when determining whether the write-ahead operation log corresponding to the second address information is stored in the first storage space based on the first filter.

[0019] In one possible implementation, to prevent the storage controller from swapping in old data that is about to be modified when swapping hot data into the cache area of ​​the acceleration device, the network processor receives a data swapping request sent by the storage controller of the storage device and can obtain the third address information and playback progress identifier carried in the data swapping request. The data swapping request is used to instruct that the data stored in the data storage area be saved to the cache area of ​​the acceleration device according to the third address information, and the playback progress identifier refers to the maximum number contained in the write-ahead operation log of the storage controller that has completed playback.

[0020] The network processor can obtain the latest operation identifier from the first filter based on the third address information. The latest operation identifier refers to the number of the write operation instruction corresponding to the last received third address information before receiving the data swap-in request. If the latest operation identifier is greater than the playback progress identifier, it means that the write-ahead operation log corresponding to the third address information is still stored in the first storage space and has not yet been played back. The data corresponding to the third address information is about to be modified. In this case, the network processor can ignore the data swap-in request and not perform the data swap-in operation to avoid swapping the old data that is about to be modified into the cache area of ​​the acceleration device. Through the above process, for the same address information, the data stored in the cache area of ​​the acceleration device can be kept consistent with the data stored in the data storage area of ​​the storage device.

[0021] In one possible implementation, the network processor can obtain the latest operation identifier from the first filter in the following manner: the network processor determines at least one second target location in the first filter based on the third address information, obtains the number of the write operation instruction stored in each of the at least one second target location, obtains at least one write operation number, and takes the smallest write operation number among the at least one write operation number as the latest operation identifier.

[0022] Through the above process, it can be ensured that the write operation number stored in each second target location is greater than the playback progress indicator. Only then will the network processor determine that the write-ahead operation log corresponding to the third address information is still stored in the first storage space and has not yet been played back, and ignore the data swap-in request. This reduces the number of misjudgments, that is, reduces the number of times the write-ahead operation log corresponding to the third address information is not stored in the first storage space but is misjudged as still being stored in the first storage space.

[0023] In one possible implementation, the log storage area of ​​the storage device includes a primary log storage area and a backup log storage area. When the network processor saves the write-ahead operation log to the log storage area in the storage device, it can save the write-ahead operation log to both the primary log storage area and the backup log storage area in the storage device, thereby improving the accuracy and validity of the stored data.

[0024] Secondly, embodiments of this application provide a data storage device, which can be disposed within the network processor of a data storage system, and an acceleration device can be connected between the host and the storage device. The device may include:

[0025] The log generation unit is used to receive the write operation instruction of the host forwarded by the interface in the acceleration device, and generate a pre-write operation log based on the write operation instruction.

[0026] The log storage unit is used to save the write-ahead operation log to the log storage area in the storage device. The write-ahead operation log is used to save the data to be written carried in the write operation instruction to the data storage area in the storage device when it is played back through the storage controller of the storage device.

[0027] In one possible implementation, the log storage unit can also be used to: send an operation execution completion notification to the host when it is determined that the write-ahead operation log has been saved.

[0028] In one possible implementation, the log storage area may include at least one storage space, and each of the at least one storage space is correspondingly provided with a filter; the write-ahead log includes first address information carried in the write operation instruction; the log storage unit is specifically used for:

[0029] The write-ahead operation log is saved to the first storage space, and the first filter corresponding to the first storage space is updated; the first storage space is any one of the at least one storage space; the updated first filter is used to characterize the write-ahead operation log corresponding to the first address information, which is saved in the first storage space and awaits playback by the storage controller of the storage device.

[0030] In one possible implementation, the first filter is stored in the acceleration device, and the device may further include a read instruction execution unit, which is used for:

[0031] Upon receiving a read operation instruction from the host, the system obtains the second address information carried in the read operation instruction; the read operation instruction is used to instruct data to be retrieved from the data storage area according to the second address information.

[0032] If, based on the first filter, it is determined that the write-ahead operation log corresponding to the second address information is stored in the first storage space, a delayed read notification is sent to the storage controller of the storage device. The delayed read notification is used to instruct the storage controller to wait for the write-ahead operation log corresponding to the second address information to be replayed before executing the read operation instruction.

[0033] In one possible implementation, the write-ahead log includes the number of the write operation instruction; the number of the write operation instruction is an incrementing number according to the time sequence in which each write operation instruction is received by the network processor; the log storage unit can be specifically used to: determine at least one first target location in the first filter according to the first address information, and save the number of the write operation instruction to each of the at least one first target location.

[0034] In one possible implementation, the apparatus may further include a data exchange unit, which may be used for:

[0035] Upon receiving a data swap-in request from the storage controller of the storage device, the third address information and playback progress identifier carried in the data swap-in request are obtained; the playback progress identifier refers to the maximum number contained in the write-ahead operation log of the storage controller that has completed playback; the data swap-in request is used to instruct that the data stored in the data storage area be saved to the cache area of ​​the acceleration device according to the third address information.

[0036] Based on the third address information, the latest operation identifier is obtained from the first filter; the latest operation identifier refers to the number of the write operation instruction corresponding to the last received third address information before the data swapping request is received;

[0037] If the latest operation identifier is greater than the playback progress identifier, the data swapping request will be ignored.

[0038] Thirdly, embodiments of this application also provide a network processor, including a processor and a power supply circuit, wherein the power supply circuit is used to supply power to the processor, and the processor is used to execute a computer program to perform any of the methods executed by the network processor described in the first aspect on data obtained through an interface.

[0039] Fourthly, embodiments of this application also provide an acceleration device, which is connected between a host and a storage device, and the acceleration device includes an interface and a network processor;

[0040] The interface is used to receive data and provide the received data to the network processor;

[0041] The network processor is used to execute computer programs to implement any of the methods performed by the network processor described in the first aspect above.

[0042] Fifthly, embodiments of this application also provide a data storage system, including a host, a storage device, and an acceleration device; the acceleration device is connected between the host and the storage device, and the acceleration device adopts the acceleration device described in the fourth aspect.

[0043] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform any of the methods provided in the first aspect above.

[0044] The technical effects that can be achieved by any of the second to sixth aspects mentioned above can be referred to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0045] Figure 1This is a schematic diagram of the structure of a data storage system in related technologies;

[0046] Figure 2 An application scenario diagram of a data storage system provided in this application embodiment;

[0047] Figure 3 This is a schematic diagram of the structure of a data storage system provided in an embodiment of this application;

[0048] Figure 4 An interactive schematic diagram of a data storage method provided in an embodiment of this application;

[0049] Figure 5 An interactive schematic diagram illustrating another data storage method provided in an embodiment of this application;

[0050] Figure 6 An interactive schematic diagram illustrating another data storage method provided in an embodiment of this application;

[0051] Figure 7 A flowchart illustrating a data storage method provided in an embodiment of this application;

[0052] Figure 8 A schematic diagram of a log storage area provided in an embodiment of this application;

[0053] Figure 9 A flowchart illustrating another data storage method provided in this application embodiment;

[0054] Figure 10 A flowchart illustrating the processing of a read operation instruction provided in an embodiment of this application;

[0055] Figure 11 A schematic diagram of a filter provided for an embodiment of this application;

[0056] Figure 12 A flowchart illustrating the processing of a data swapping request provided in an embodiment of this application;

[0057] Figure 13 A structural block diagram of a data storage device provided in an embodiment of this application;

[0058] Figure 14 A structural block diagram of a chip provided in an embodiment of this application;

[0059] Figure 15 This is a structural block diagram of an acceleration device provided in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application.

[0061] Before introducing the specific solutions provided in the embodiments of this application, some terms used in this application will be explained to facilitate understanding by those skilled in the art, but the terms used in this application are not limited.

[0062] (1) Acceleration devices: These are used to offload some functions from the host computer. For example, data processing functions that are unsuitable for the host computer to handle can be offloaded to acceleration devices, freeing up the host's computing power. Acceleration devices may include, but are not limited to, computing units with offloading capabilities such as data processing units (DPUs), infrastructure processing units (IPUs), system-on-chips (SoCs), iNICs, or smartNICs. iNICs or smartNICs can be understood as intelligent network interface cards (NICs).

[0063] (2) Network processor (NP): Primarily used to handle network-related services, such as data transmission. NP supports programmable architecture and has remote direct memory access (RDMA) capability.

[0064] (3) File System: A software system used to manage and store file information. It organizes and allocates space on file storage devices, and is responsible for file storage, protection, and retrieval. The file system's data storage area can include a metadata storage area and a data storage area. The data storage area is responsible for storing files or data within files; in the file data area, files are stored as file data blocks. The metadata storage area supports the file system architecture and stores metadata such as file indexes and validity, as well as the file system's own attribute data. In other words, data in the file system is divided into data of different data types and data of different metadata types. Data refers to the actual data in the file, i.e., the file's content; while metadata refers to system data used to describe the characteristics of a file, such as access permissions, file owner, and the distribution information of file data blocks. Distribution information can include inodes. To operate on a file in the file system, you must first obtain its metadata in order to locate the file and obtain its content or related attributes.

[0065] In this application embodiment, "multiple" refers to two or more. Therefore, in this application embodiment, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, such as one, two, or more. For example, "including at least one" means including one, two, or more, and it does not limit which ones are included. For example, including at least one of A, B, and C, then it could include A, B, C, A and B, A and C, B and C, or A and B and C. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0066] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects, and are not used to limit the order, sequence, priority or importance of multiple objects.

[0067] Data storage systems can be applied to scenarios such as cloud storage or cloud computing. In data storage systems, to conserve the computing power of the host machine, acceleration devices can be used to offload the host's data access functions. For example, Figure 1 The data storage system shown includes a host 100, an acceleration device 200, and a storage device 300. The acceleration device 200 is connected between the host 100 and the storage device 300, and the host 100 can access the storage device 300 through the acceleration device 200. The host 100 and the acceleration device 200 can be a component of a physical machine, and the physical machine and the storage device 300 can communicate remotely via a network.

[0068] Host 100 can be understood as the core processor of the physical machine, serving as its computational and control core. The physical machine can be, but is not limited to, a physical server in a cloud computing cluster, a computing device in a computing device cluster, or a server in a network management center. Host 100 can receive data input by users through a client and process that data.

[0069] To persistently store data, the storage device 300 in the data storage system can be located outside the host 100 and exchange data with the host 100 via a network. The storage device 300 may include a storage controller 310 and a disk enclosure 320. The storage controller 310 can be used to manage and control the disk enclosure 320. The storage controller 310 and the disk enclosure 320 can be connected via a bus or communicate remotely via a network.

[0070] Acceleration device 200 is used to offload some functions of host 100. For example, acceleration device 200 can offload the data access function of host 100. By offloading some functions of host 100, acceleration device 200 can dedicate host 100 to management and control functions, thereby improving the performance of host 100.

[0071] like Figure 1 As shown, in related technologies, the acceleration device 200 may include a main processor 210 and an application-specific integrated circuit (ASIC) chip 220. The main processor 210 may be implemented using a central processing unit (CPU) or other general-purpose processor. The ASIC chip 220 may include a hardware queue 221 and a network engine 222, wherein the network engine 222 can be understood as a network interface, and the network engine 222 may be implemented using a remote memory access over converged Ethernet (RoCE) interface.

[0072] For example, Figure 1 The data storage system shown can be used to handle file input / output (I / O) operations in the file system. Host 100 can send file operation commands to hardware queue 221 in acceleration device 200 via a queue channel. The main processor 210 of acceleration device 200 reads the file operation instructions from hardware queue 221, executes the file I / O operation, generates data access instructions based on the file I / O operation, and then sends the data access instructions to storage controller 310 via network engine 222 in acceleration device 200. Storage controller 310 interacts with disk enclosure 320 to execute the data access instructions. After the data access instructions are executed, storage controller 310 returns an access completion message to acceleration device 200. Acceleration device 200 receives the access completion message from storage controller 310 via network engine 222 and transmits it to main processor 210. Main processor 210 processes the access completion message and then notifies host 100 of the file operation instruction execution completion via hardware queue 221 in ASIC chip 220.

[0073] During the aforementioned file I / O operation process, the main processor 210 mainly handles the file I / O operations of the acceleration device 200, which consumes the computing resources of the main processor 210 and affects the performance of the acceleration device 200. Furthermore, there are four interactions between the main processor 210 and the ASIC chip 220 of the acceleration device 200, which increases the latency of the file I / O operation.

[0074] Based on this, embodiments of this application provide a data storage system. Exemplarily, in some application scenarios, such as... Figure 2 As shown, the data storage system may include multiple physical machines, such as physical machine 410, physical machine 420, physical machine 430, etc. Each physical machine may include a host and an acceleration device, which can communicate remotely with the storage controller and disk enclosure via a network. Each physical machine can interact with the storage controller or disk enclosure through the acceleration device to exchange data or signaling. Figure 2 This example uses two storage controllers and two disk enclosures, such as storage controller 510, storage controller 520, disk enclosure 530, and disk enclosure 540. In actual use, the number of storage controllers and disk enclosures can be more than two or less than two. The storage controllers and disk enclosures can be located in a single storage device or can be set up separately and communicate remotely via a network. The storage controllers are used to manage and control the disk enclosures.

[0075] Figure 3 Let's take the interaction between a physical machine and a storage device as an example to illustrate, such as... Figure 3 As shown, the physical machine 600 may include a host 610 and an acceleration device 620 connected to the host 610. In some embodiments, the acceleration device 620 may be directly inserted into a slot on the motherboard of the host 610 and exchange data or signaling with the host 610 via the PCIe bus. It should be noted that the PCIe bus can be replaced with a bus based on the compute express link (CXL), Universal Serial Bus (USB) protocol, or other protocols.

[0076] Storage device 700 may include storage controller 710 and multiple disk enclosures connected to storage controller 710, such as disk enclosure 720, disk enclosure 730, etc. Storage controller 710 can be used to manage and control multiple disk enclosures. Figure 3 This example uses two disk frames. In actual use, the number of disk frames can be more than two or less.

[0077] The disk enclosure can be a solid-state drive (SSD), which is a hard drive made of solid-state electronic storage chip arrays. It can include a control unit and storage units. The control unit can receive instructions from the network processor or storage controller to manage and control the storage units. The storage units can be flash memory chips or dynamic random access memory (DRAM) chips.

[0078] like Figure 3As shown, the acceleration device 620 may include a main processor 621, a network processor 622, and internal memory 623. The main processor 621 may be a CPU or a programmable logic device (PLD) chip. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The main processor 621 is used to run the operating system of the acceleration device 620 and the software programs running based on the operating system. The network processor 622 is used to receive and process file I / O operations sent by the host 610.

[0079] The cache area in the internal memory 623 can be used to store the data buffer and metadata buffer corresponding to the file system. The area used to store the data buffer can be called the data cache, and the area used to store the metadata buffer can be called the metadata cache. In some embodiments, the data cache stores data items, which can be used to store the storage address of data in the disk frame. The metadata cache can store the metadata of the file system. The metadata can adopt a tree structure, including multi-level parent node information and child node information.

[0080] In this embodiment, the host 610 can send file I / O operation instructions to the acceleration device 620. The acceleration device 620 receives the operation instructions sent by the host 610 through an interface and forwards them to the network processor 622 in the acceleration device. If the received operation instruction is a write operation instruction, the network processor 622 can generate a write-ahead log based on the write operation instruction. The write operation instruction refers to an operation instruction carrying data to be written, which may include, but is not limited to, data modification operation instructions. The write-ahead log can also be called a write-ahead logging (WAL). After generating the write-ahead log, the network processor 622 can save the write-ahead log to the log storage area in the storage device 700. The write-ahead log is used to save the data to be written carried in the write operation instruction to the data storage area in the storage device when played back through the storage controller of the storage device 700. The log storage area and the data storage area can be stored in the disk frame of the storage device 700. The log storage area is used to store the write-ahead log, also known as the WAL area; the data storage area is used to store file system data, also known as the persistent logging (PLOG) area.

[0081] For example, Figure 3 The disk frame 720 shown includes a log storage area 721 and a data storage area 722. The disk frame 730 includes a log storage area 731 and a data storage area 732. In some embodiments, log storage areas 721 and 731 can be used to store different write-ahead operation logs. In other embodiments, to ensure the reliability of stored data, log storage areas 721 and 731 can be mutually primary and backup, meaning that the same write-ahead operation log can be stored in both log storage areas 721 and 731. In other embodiments, the primary and backup WAL areas can also be located within the same disk frame.

[0082] In this embodiment, the acceleration device processes the received write operation instructions through a network processor, which saves the computing resources of the acceleration device's main processor, improves the performance of the acceleration device, and eliminates the need for multiple interactions between the processor and the network processor, thereby shortening the latency of I / O operations. The network processor generates a write-ahead log based on the received write operation instructions and saves the write-ahead log to the log storage area in the storage device. These two steps are relatively simple and fast, thus saving time. The complex process of replaying the write-ahead log is executed by the storage controller, and the replay can be completed in batches within the storage controller without affecting the latency of the acceleration device's I / O operations.

[0083] This application embodiment also provides a method that can be applied to Figure 2 and Figure 3 The data storage system shown illustrates a data storage method. In some embodiments of the data storage method, the file I / O operation instructions received by the network processor from the host can be categorized into two types based on the operation type: operation instructions with high execution frequency and high requirements for processor performance and latency can be classified into the first type, such as file creation, modification, read I / O, and write I / O; operation instructions with low execution frequency, or relatively low requirements for processor performance and latency, can be classified into the second type, such as renaming, mounting, and initialization. Different processing methods can be adopted to address the different requirements of different operation instructions.

[0084] For example, when a host needs to operate on the file system, it can send a file I / O operation command to the acceleration device. After receiving the command, the interface in the acceleration device forwards it to the network processor. Upon receiving the command, the network processor performs semantic parsing to determine the corresponding operation type and processes it accordingly.

[0085] Figure 4 An exemplary flowchart illustrates the interaction between various devices when processing a second type of operation instruction in a data storage method. For example... Figure 4 As shown, the method may include the following steps:

[0086] S401, the host sends an operation command to the acceleration device.

[0087] S402, the acceleration device determines the operation type of the operation instruction as a second type of operation instruction through the network processor.

[0088] After receiving an operation command from the host, the interface in the acceleration device forwards the command to the network processor within the device. The network processor can determine the operation type corresponding to the command through semantic parsing. For example, if the network processor determines through semantic parsing that the command is an initialization command, it can then classify the command as a type II operation command.

[0089] S403, the acceleration device sends operation commands to the storage controller.

[0090] S404, Storage Controller processes operation commands.

[0091] In some embodiments, the network processor of the acceleration device can forward the second type of operation instruction to the storage controller in the storage device for processing. For example, if the received second type of operation instruction is an initialization operation instruction, the storage controller executes the initialization operation instruction to initialize the disk frame indicated by the initialization operation instruction.

[0092] In other embodiments, since the second type of I / O operations are executed at a low frequency and have relatively low requirements for processor performance, the network processor can forward the second type of operation instructions to the main processor in the acceleration device, where the main processor executes the instructions. For example, the network processor can forward received initialization operation instructions to the main processor in the acceleration device for processing. The main processor in the acceleration device executes the initialization operation instructions, determines the disk frames that need to be initialized, and initializes the disk frames through the storage controller.

[0093] Figure 5 An exemplary flowchart illustrates the interaction between various devices when processing a read operation instruction in a first type of operation instruction in a data storage method. For example... Figure 5 As shown, the method may include the following steps:

[0094] S501, the host sends operation commands to the acceleration device.

[0095] S502, the acceleration device determines the operation type of the operation instruction as a read operation instruction through the network processor, and obtains the address information carried in the read operation instruction.

[0096] After receiving an operation command from the host, the interface in the acceleration device forwards the command to the network processor within the acceleration device. The network processor, through semantic parsing, determines that the operation type corresponding to the command is a read operation command. Specifically, the read operation command instructs the retrieval of data from the data storage area of ​​the storage device based on the address information carried by the read operation command.

[0097] S503, the acceleration device uses the network processor to search the acceleration device's cache area based on the address information, and determines that the cache area does not contain the address information.

[0098] S504, the acceleration device sends the read operation command to the storage controller.

[0099] The address information can be the logical address of the data to be read. After obtaining the logical address carried in the read operation instruction, the network processor in the acceleration device checks whether the cache area of ​​the acceleration device contains the logical address. If it does, the data corresponding to the logical address can be retrieved from the cache area; otherwise, the network processor in the acceleration device can forward the read operation instruction to the storage controller in the storage device for processing. The storage controller can read data from the data storage area in the disk frame according to the address information carried in the read operation instruction. In some embodiments, the network processor in the acceleration device can also send the read operation instruction to the main processor in the acceleration device for processing. The main processor in the acceleration device can read data from the data storage area in the disk frame according to the address information carried in the read operation instruction.

[0100] In some embodiments, read operation instructions can be divided into read data operation instructions and read metadata operation instructions, depending on the data type to be read. Read data operation instructions are used to read data of a specific data type; read metadata operation instructions are used to read data of a specific metadata type. For read data operation instructions, the network processor can, after obtaining the logical address, query whether the data cache in the cache area contains that logical address. For read metadata operation instructions, the network processor can, after obtaining the logical address, query whether the metadata cache in the cache area contains that logical address.

[0101] The second type of operation instructions and read operation instructions mentioned above can be collectively referred to as non-modification operation instructions. Non-modification operation instructions can include any operation instruction except for write operation instructions. Compared to write operation instructions, non-modification operation instructions can refer to operation instructions that do not carry data to be written. Considering that non-modification operation instructions have lower latency requirements and consume less processor computing resources, they will not significantly affect the performance of the acceleration device. Therefore, after receiving non-modification operation instructions from the host, the network processor can either send the non-modification operation instructions to the main processor of the acceleration device for processing, or send them to the storage controller of the storage device for processing.

[0102] Figure 6 An exemplary flowchart illustrates the interaction between various devices when processing a write operation instruction in a first type of operation instruction in a data storage method. For example... Figure 6 As shown, the method may include the following steps:

[0103] S601, the host sends operation commands to the acceleration device.

[0104] S602, the acceleration device determines the operation type of the operation instruction as a write operation instruction through the network processor, and generates a write-ahead operation log based on the address information and the data to be written carried in the write operation instruction.

[0105] After receiving an operation command from the host, the interface in the acceleration device forwards the command to the network processor within the acceleration device. The network processor, through semantic parsing, determines that the operation type corresponding to the command is a write operation command. Specifically, the write operation command instructs the writing or updating of data in the data storage area of ​​the storage device based on the address information carried by the command. The address information can be the logical address corresponding to the data to be written.

[0106] The network processor can obtain the logical address and the data to be written carried in the write operation instruction, and generate a write-ahead log containing the logical address and the data to be written.

[0107] In some embodiments, write operation instructions can be divided into write data operation instructions and write metadata operation instructions, depending on the data type of the data to be written. Write data operation instructions are used to write data of a specific data type; write metadata operation instructions are used to write data of a specific metadata type.

[0108] Before generating a write operation log, the network processor can obtain the logical address carried in the write operation instruction and check whether the logical address is stored in the acceleration device's data cache. In one embodiment, if the logical address is stored in the data cache, it means that the associated data item of the write operation instruction is stored there. The network processor can clear the associated data item of the write operation instruction, i.e., clear the associated data item corresponding to the logical address, without storing the data to be written and the logical address carried in the write operation instruction in the data cache. This avoids frequent flushing of the data cache due to a large number of write operations, reducing resource consumption. In another embodiment, if the logical address is stored in the data cache, the network processor can update the associated data item corresponding to the logical address based on the data to be written. If the logical address is not stored in the data cache, the network processor can create a new data item in the data cache to store the data to be written in the write operation instruction and the corresponding logical address.

[0109] For write metadata operation instructions, before generating the write-ahead operation log, the network processor can obtain the data to be written and its corresponding logical address carried in the write metadata operation instruction, and add the data to be written and its corresponding logical address to the metadata cache of the acceleration device. The data to be written is metadata type data. Once the metadata cache in the acceleration device is full, the acceleration device can save the contents of the metadata cache to the storage area in the storage controller used to store file system metadata.

[0110] S603, the acceleration device writes the pre-written operation log to the disk frame.

[0111] S604, the acceleration device sends an operation completion notification to the host.

[0112] The acceleration device can save the generated write-ahead operation log to the log storage area within the disk frame via a network processor, and send an operation completion notification to the host after confirming that the write-ahead operation log has been saved. For example, the network processor can write the write-ahead operation log to the disk frame via an RDMA channel. The RDMA channel is implemented based on the RDMA protocol. Through the RDMA protocol, the network processor transmits the write-ahead operation log to the disk frame, the disk frame saves the received write-ahead operation log to the log storage area, and sends an ACK message back to the network processor after saving. The ACK message is an acknowledgment message used to notify the network processor that the write-ahead operation log has been written successfully. Upon receiving the ACK message from the disk frame, the network processor confirms that the write-ahead operation log has been saved successfully and sends an operation completion notification to the host. The write-ahead operation log is used to save the data to be written carried in the write operation instruction to the data storage area within the disk frame during playback via the storage controller. The data storage area can include a metadata area and a file data area. If the data to be written is metadata type data, it can be written to the metadata area; if the data to be written is data of a different data type, it can be written to the file data area.

[0113] The processes described above, including generating write-ahead logs based on write operation instructions and writing these logs to the storage device's disk frame, are all executed by the network processor of the acceleration device. After completion, the network processor directly returns an operation completion notification to the host. This process eliminates the need for multiple interactions between the network processor and the main processor, reducing the number of interactions, effectively decreasing the latency of executing operation instructions, achieving a fast I / O access path, and improving the overall performance of the data storage system.

[0114] The log storage area can be called the WAL area. In some embodiments, to ensure the accuracy and validity of the stored data, a primary WAL area and a backup WAL area can be set up in the disk frame, and write-ahead logs can be saved to both the primary WAL area and the backup WAL area simultaneously. The primary WAL area and the backup WAL area can be located in the same disk frame or in different disk frames.

[0115] S605, the storage controller sends a read log notification to the disk frame.

[0116] S606, the disk enclosure sends a write-ahead operation log to the storage controller.

[0117] When the storage controller performs log replay, it can retrieve the write-ahead operation log from the disk frame. The storage controller sends a read log notification to the disk frame, and the disk frame sends the write-ahead operation log back to the storage controller based on the received read log notification. When a primary WAL area and a backup WAL area exist, the storage controller can specify to read the write-ahead operation log from either the primary WAL area or the backup WAL area. After replay is complete, the write-ahead operation log stored in both the primary and backup WAL areas can be cleared simultaneously. For example, if the storage controller specifies to read the write-ahead operation log from the primary WAL area, and the log stored in the primary WAL area is incorrect and cannot be read normally, the write-ahead operation log can be read from the backup WAL area.

[0118] S607, the storage controller replays the received write-ahead operation log.

[0119] The write-ahead log includes the data to be written carried by the write operation instruction and the logical address corresponding to the data to be written. By replaying the write-ahead log, the storage controller can save the data to be written carried by the write operation instruction to the data storage area in the disk frame according to the logical address carried by the write operation instruction. For example, the storage controller determines the physical address corresponding to the logical address in the data storage area based on the logical address carried by the write operation instruction, and writes the data to be written to the data storage area based on the determined physical address.

[0120] In the above embodiments, the data storage method for accelerating device execution may include, for example: Figure 7 The following steps are shown:

[0121] In S701, the interface in the acceleration device receives write operation commands sent by the host and forwards the write operation commands to the network processor in the acceleration device.

[0122] S702, the network processor in the acceleration device generates a write-ahead log based on write operation instructions.

[0123] The write-ahead log includes the data to be written carried by the write operation instruction and the address information corresponding to the data to be written.

[0124] In the S703, the network processor in the acceleration device saves the write-ahead operation log to the log storage area in the storage device.

[0125] The write-ahead log is used to save the data to be written carried in the write operation instruction to the data storage area in the storage device when it is played back through the storage controller of the storage device.

[0126] If the network processor determines that the write-ahead operation log has been saved, it sends an operation completion notification to the host.

[0127] Processing received write operation commands through a network processor can conserve computing resources of the acceleration device's main processor, thereby improving the device's performance. The network processor generates a write-ahead log based on the received write operation commands and saves this log to the log storage area in the storage device. These two steps are relatively simple and fast, thus saving time. The complex process of replaying the write-ahead log is performed by the storage controller, and this replay can be done in batches within the storage controller without increasing the latency of the acceleration device's I / O operations.

[0128] In some embodiments, the interface in the acceleration device receives a read operation command sent by the host. The network processor in the acceleration device can then send the read operation command to the storage controller, which reads the required data from the data storage area of ​​the disk frame according to the address information carried in the read operation command. The address information carried in the read operation command can be called read operation address information. Considering that when the storage controller executes the read operation command, the data to be written corresponding to the read operation address information may still be stored in the write-ahead log, i.e., the write-ahead log is still in the log storage area and has not yet been played back, for example, at a first moment, the host sends a first operation command to the acceleration device. The first operation command is a write operation command, instructing the writing of first data to address a. The network processor receives the first operation command, generates a first write-ahead log based on the first operation command, and writes the first write-ahead log to the log storage area of ​​the disk frame. The first write-ahead log includes the first data and the address a corresponding to the first data. Immediately following, at the second moment, the host sends a second operation command to the acceleration device. This second operation command is a read operation command, instructing the read of data from address a. However, since the first write-ahead log corresponding to address a is still in the log storage area and has not yet been replayed, the first data corresponding to address a has not yet been written to the data storage area. If the storage controller executes the read operation command at this time, it may fail to read the data, or it may read incorrect data.

[0129] To address the issue that the storage controller might read data that has not yet been written to the data storage area, this embodiment divides the log storage area within the disk frame into at least one storage space. Each storage space stores write-ahead logs generated based on write operation instructions. Each storage space can store one or more write-ahead logs, and each storage space can be referred to as a data file space (segment). Each storage space is assigned a filter to determine whether a specific address is included in the write-ahead logs stored in the corresponding storage space that have not yet been replayed. For example... Figure 8 As shown, the log storage area 810 in disk frame 800 may include multiple storage spaces. Figure 8Only three storage spaces are shown: storage space 811, storage space 812, and storage space 813. The first filter is the filter corresponding to storage space 811, the second filter is the filter corresponding to storage space 812, and the third filter is the filter corresponding to storage space 813. These filters can be Bloom filters, sparse filters, or other types of filters.

[0130] In some embodiments, such as Figure 9 As shown, the process by which the network processor in the acceleration device writes a write-ahead operation log to the disk frame may include the following steps:

[0131] S901 generates a write-ahead log based on the received write operation instruction.

[0132] Upon receiving a write operation command, the network processor can generate a write-ahead log based on the data to be written and the first address information carried in the write operation command. The write-ahead log includes the first address information carried in the write operation command, where the first address information can be the logical address corresponding to the data to be written.

[0133] S902 saves the write-ahead operation log to the current storage space in the log storage area.

[0134] The current storage space can be any of multiple storage spaces. The network processor sequentially writes write-ahead operation logs to each storage space in the log storage area. For example, the network processor first writes... Figure 8 The write-ahead operation log is written into storage space 811, which is the current storage space. If storage space 811 is full after writing 30 write-ahead operation logs, the network processor can write the 31st write-ahead operation log into storage space 812 after it is generated. Storage space 812 is the current storage space at this time. After storage space 812 is full, newly generated write-ahead operation logs can be written into storage space 813, which is the current storage space at this time.

[0135] S903 updates the current filter corresponding to the current storage space based on the first address information in the write-ahead operation log.

[0136] The updated current filter is used to indicate that the write-ahead operation log corresponding to the first address information is stored in the current storage space and awaits replay through the storage controller.

[0137] For example, a network processor can process the first address information using a set of hash functions to determine a set of hash values. These hash values ​​indicate the corresponding positions of the first address information in the filter, allowing updates to the elements at those positions. The number of hash values ​​obtained is the same as the number of hash functions. If there are three hash functions, processing the first address information will yield three hash values. Based on these three hash values, the three corresponding positions of the first address information in the filter can be determined. For instance, assuming the filter contains 64 elements from 0 to 63, each occupying 1 bit, this can be understood as the filter containing 64 positions. Initially, all 64 positions in the filter are 0. If processing the first address information yields three hash values ​​of 8, 15, and 19, this indicates that the first address information corresponds to positions 8, 15, and 19 in the filter, respectively. The elements at positions 8, 15, and 19 can then be set to 1. Conversely, when the elements at positions 8, 15, and 19 are all 1, it means that the write-ahead log corresponding to the first address information is stored in the current storage space. At this time, the data to be written corresponding to the first address information has not yet been written to the data storage area.

[0138] The current filter corresponding to the current storage space can be stored in the internal memory of the acceleration device. Once the current storage space is full, the network processor saves the contents of the current filter to the disk frame. For example, when the disk frame detects that the current storage space is full, it can send a message indicating that the storage space is full to the network processor. Upon receiving this message, the network processor determines that the current storage space is full and saves the contents of the current filter to the disk frame, which then associates the current filter with the corresponding storage space. For example, Figure 8 Storage spaces 811 and 812 are full. The first filter corresponding to storage space 811 and the second filter corresponding to storage space 812 are stored in the disk frame. However, storage space 813 is not yet full, meaning that the network processor is still writing write-ahead operation logs into storage space 813. Storage space 813 is the current storage space, and the third filter corresponding to storage space 813 is the current filter. The contents of the third filter are still stored in the internal memory of the acceleration device and are not stored in the disk frame. Therefore, the third filter in the disk frame is represented by a dashed box.

[0139] The above describes the process by which the network processor writes write-ahead logs to the disk frame. For the write-ahead logs stored in the disk frame, the storage controller can sequentially replay the write-ahead logs stored in each storage space. When all the write-ahead logs stored in a certain storage space have been replayed, the storage controller can control the disk frame to clear that storage space and clear the contents stored in the corresponding filter, resetting all elements in the filter to 0.

[0140] In some embodiments, such as Figure 10 As shown, the process of the network processor in the acceleration device processing read operation commands sent by the host may include the following steps:

[0141] S1001, retrieve the second address information carried in the read operation instruction.

[0142] When a network processor receives an operation command from a host, it can determine the operation type through semantic parsing. If the received operation command is determined to be a read operation command, the second address information carried in the read operation command can be obtained. The read operation command instructs the retrieval of data from the data storage area based on the second address information. The second address information can be the logical address of the data to be read.

[0143] S1002, determine whether the cache area contains the second address information; if yes, proceed to step S1003; if no, proceed to step S1004.

[0144] S1003 reads the data corresponding to the second address information from the cache area and sends the read data to the host.

[0145] The network processor searches the cache area of ​​the acceleration device based on the second address information to determine whether the cache area contains the second address information. If the cache area of ​​the acceleration device contains the second address information, it can retrieve the data corresponding to the second address information from the cache area and send it to the host.

[0146] S1004. Based on the current filter, determine whether the write-ahead operation log corresponding to the second address information is stored in the current storage space; if yes, proceed to step S1006; if no, proceed to step S1005.

[0147] If the cache area of ​​the acceleration device does not contain the second address information, it means that data needs to be read from the data storage area of ​​the disk frame. In this case, the network processor can determine whether the write-ahead operation log corresponding to the second address information is stored in the current storage space based on the current filter stored in the acceleration device.

[0148] For example, the network processor can process the second address information using a set of hash functions to obtain a set of hash values, i.e., the hash values ​​corresponding to the second address information. The network processor can then determine, within the current filter, whether the element at the position indicated by the hash value corresponding to the second address information is 1.

[0149] Considering that processing two different address information using a single hash function might yield the same hash value, for example, if the write-ahead log (WAP) corresponding to address b is stored in the current storage space, while the WAP log corresponding to address c is not, processing both addresses b and c with the same hash function will result in the same hash value, where the element at the corresponding position is 1. Therefore, based on this hash value, it can be concluded that the WAP log corresponding to address c is stored in the current storage space, leading to a false positive. To reduce false positives, multiple hash functions can be used to process the same address information, yielding multiple hash values. If all elements at the positions indicated by multiple hash values ​​are 1, then it can be determined that the WAP log corresponding to that address information is stored in the current storage space. If at least one element at the position indicated by multiple hash values ​​is not 1, then it can be determined that the WAP log corresponding to that address information is not stored in the current storage space.

[0150] Assuming the second address information is processed, the resulting hash value includes three values. If all elements at the positions indicated by the three hash values ​​corresponding to the second address information are 1, then it can be determined that the write-ahead log corresponding to the second address information is stored in the current storage space. If one or more elements at the positions indicated by the three hash values ​​corresponding to the second address information are not 1, then it can be determined that the write-ahead log corresponding to the second address information is not stored in the current storage space.

[0151] S1005 sends the read operation command to the storage controller of the storage device.

[0152] When the network processor determines that the write-ahead log corresponding to the second address information is not stored in the current storage space, it can send a read operation command to the storage controller of the storage device. This allows the storage controller to determine, based on the filters stored in the disk frame, whether the write-ahead log corresponding to the second address information is stored in the log storage area of ​​the disk frame, and wait for playback through the storage controller.

[0153] For example, suppose the current filter stored in the acceleration device is Figure 8The third filter corresponds to storage space 813, which is the current storage space. If the network processor determines that the write-ahead log corresponding to the second address information is not stored in storage space 813, it can send a read operation command to the storage controller. The storage controller can then retrieve the filters stored in the disk frame, such as... Figure 8 The first filter and the second filter are shown. The storage controller can determine, based on the first filter, whether the write-ahead log corresponding to the second address information is stored in storage space 811, and based on the second filter, whether the write-ahead log corresponding to the second address information is stored in storage space 812. If the write-ahead log corresponding to the second address information is not stored in storage space 811, and is not stored in storage space 812, the storage controller can execute the read operation instruction to read data from the data storage area based on the second address information. Otherwise, the storage controller can wait for the write-ahead log corresponding to the second address information to be replayed before executing the read operation instruction.

[0154] S1006, Send a delayed read notification to the storage controller of the storage device.

[0155] The delayed read notification may include the aforementioned read operation instruction. The delayed read notification instructs the storage controller to wait for the write-ahead log corresponding to the second address information to complete its playback before executing the read operation instruction.

[0156] In some embodiments, a delayed read notification can be a read operation instruction with a delayed read flag added. For example, if the network processor determines that the write-ahead log corresponding to the second address information is stored in the current storage space and has not yet been replayed, it indicates that the read operation instruction conflicts with a previous write operation instruction, or that there is a dependency relationship. In this case, the network processor can add a delayed read flag to the read operation instruction and send the read operation instruction with the delayed read flag added to the storage controller.

[0157] When the storage controller receives a delayed read notification from the network processor, it can wait for the write-ahead log corresponding to the second address information to be replayed before executing the read operation instruction.

[0158] In some embodiments, each operation instruction received by the network processor has a unique number. This number can be an incrementing sequence number, also known as a sequence number, based on the chronological order in which the network processor receives each operation instruction. The write-ahead log generated based on the write operation instructions may include the corresponding write operation instruction number. The delayed read notification received by the storage controller from the network processor includes the read operation instruction and its number. When the storage controller performs log replay, it replays the write operation instructions sequentially according to their chronological order. The storage controller can determine the number of the write operation instruction contained in the write-ahead log being replayed; this number can be called the replay progress identifier. The storage controller compares the replay progress identifier with the read operation instruction number in the delayed read notification. If the replay progress identifier is less than the read operation instruction number, it indicates that the write-ahead log corresponding to the second address information may not have been replayed yet, and the storage controller continues to wait. If the playback progress indicator is greater than the number of the read operation instruction, it means that the write-ahead logs corresponding to all write operation instructions before the read operation instruction have been played back, and the write-ahead log corresponding to the second address information must have been played back. Then the storage controller can execute the read operation instruction and read data from the data storage area based on the second address information.

[0159] In other embodiments, the length of each element at each position in the filter can be increased; for example, the length of each element can be increased from 1 bit to 4 bytes (32 bits) or 8 bytes (64 bits). Each operation instruction received by the network processor has a unique number, and the write-ahead log generated based on the write operation instruction can include the corresponding write operation instruction number. In practical use, the length of each element at each position in the filter can be determined according to actual needs. The longer the element is, the larger the maximum number it can store, which should be greater than or equal to the number of operation instructions sent by the host to the network processor within a set period. At the beginning of the next period, the numbering of the operation instructions sent by the host to the network processor starts from 0 and is reset.

[0160] When updating the current filter based on the first address information in the write-ahead operation log, the first target location can be determined within the current filter based on the first address information, and the write operation instruction number can be saved to the first target location in the current filter. For example, a set of hash functions can be used to process the first address information to obtain one or more hash values. Based on these hash values, one or more first target locations can be determined within the current filter, and the write operation instruction number corresponding to the first address information can be saved to each first target location. Saving the write operation instruction number corresponding to the first address information to the first target location significantly reduces the probability of false positives compared to setting the element in the first target location to 1.

[0161] For example, such as Figure 11 As shown, assuming that three hash functions are used to process the first address information, the three hash values ​​obtained are 8, 15, and 19 respectively. This means that the first address information corresponds to positions 8, 15, and 19 in the current filter, respectively. The write operation instruction number S11 can be saved to positions 8, 15, and 19 respectively. Other positions in the current filter may contain other instruction numbers, or they may not contain any instruction numbers. If a position does not contain any instruction number, the element corresponding to that position can be an initial value of 0. Saving the write operation instruction number S11 in multiple positions is also to avoid misjudgment, as explained above, and will not be repeated here.

[0162] When the network processor determines that the write-ahead operation log corresponding to the second address information is stored in the current storage space, it can obtain the write operation instruction number corresponding to the second address information from the current filter based on the second address information. This number can be called the conflict operation number. The conflict operation number can be included in the delayed execution notification sent to the storage controller.

[0163] When the storage controller performs log replay, it can determine the number of the write operation instruction contained in the write-ahead log being replayed; this number can be called the replay progress identifier. After each write-ahead log is replayed, the storage controller compares the replay progress identifier with the conflict operation identifier. If the replay progress identifier and the conflict operation identifier are the same, it means that the write-ahead log corresponding to the second address information has been replayed successfully, and the storage controller can execute the read operation instruction to read data from the data storage area based on the second address information. If the replay progress identifier and the conflict operation identifier are different, it means that the write-ahead log corresponding to the second address information has not yet been replayed, and the storage controller continues to wait.

[0164] In the above embodiments, by dividing the log storage area into one or more storage spaces and setting a filter for each storage space, the system determines whether the write-ahead log corresponding to the address information carried by the read operation instruction is located in the storage space and has not yet been replayed, thereby avoiding the problem that the storage controller cannot read data or reads incorrect data when executing the read operation instruction.

[0165] To facilitate faster data access by the host and more efficient utilization of the cache area in the acceleration device, the storage controller can move frequently accessed hot data from the host to the cache area in the acceleration device. The storage controller can identify hot data; upon detection, it can read the hot data from the data storage area and write it to the cache area in the acceleration device. Alternatively, it can obtain the address used to store the hot data from the data storage area from the disk frame and write that address to the cache area in the acceleration device.

[0166] During data swapping, the storage controller may encounter the following problems: In some embodiments, when the storage controller determines that data A is hot data and retrieves data A from the data storage area, the network processor in the acceleration device may have already saved the write-ahead log corresponding to the write operation instruction that modifies data A to the log storage area, meaning data A is about to be modified. In this case, the storage controller cannot know that data A is about to be modified and still writes data A to the cache area in the acceleration device, resulting in unmodified old data being swapped into the cache area of ​​the acceleration device. In other embodiments, when the storage controller determines that data A is hot data and retrieves the address ADD1 for storing data A from the data storage area, the network processor may have already saved the write-ahead log corresponding to the write operation instruction that modifies data A to the log storage area. At this point, the storage controller is unaware that data A is about to be modified and will still write the address ADD1 of data A into the cache area of ​​the acceleration device. When the write-ahead operation log corresponding to the write operation instruction that modifies data A is replayed, a new address ADD2 may be configured for the modified data A, and the modified data A may be saved to the new address ADD2. As a result, when the host or acceleration device reads data according to the address ADD1 saved in the data cache, it reads the old data that has not been modified. It is equivalent to swapping the old data that has not been modified into the cache area of ​​the acceleration device.

[0167] To address the issue of storage controllers swapping unmodified old data into the cache area, in some embodiments, each write operation instruction received by the network processor in the acceleration device has a unique number. The number of each instruction can be an ascending sequence of numbers representing the times in which the network processor received each write operation instruction. The write-ahead log generated based on the write operation instructions can include the corresponding write operation instruction numbers. When the network processor updates the current filter based on the first address information in the write-ahead log, it can determine one or more first target locations within the current filter based on the first address information and save the write operation instruction number corresponding to the first address information to each first target location. The specific execution process has been described above and will not be repeated here.

[0168] During the data swapping process, the network processor in the acceleration device can perform actions such as... Figure 12 The following steps are shown:

[0169] S1201, Receive a data swap-in request sent by the storage controller of the storage device, and obtain the third address information and playback progress identifier carried in the data swap-in request.

[0170] For example, when the storage controller determines that data A stored in the data storage area is hot data, it can send a data swap-in request to the network processor of the acceleration device. The data swap-in request may include third address information corresponding to data A, or it may include data A and its corresponding third address information. The data swap-in request instructs the network processor to save the data stored in the data storage area to the cache area of ​​the acceleration device based on the third address information. The third address information may be the logical address corresponding to data A, which can be determined based on the physical address of data A used to store it in the data storage area.

[0171] The data swap-in request may also include a playback progress identifier, which refers to the maximum number S_MAX contained in the write-ahead operation log that the storage controller has completed playback of, that is, the number of the write operation instruction contained in the last write-ahead operation log that the storage controller has completed playback of.

[0172] S1202, based on the third address information, obtain the latest operation identifier from the current filter.

[0173] The latest operation identifier refers to the number of the write operation instruction corresponding to the last third address information received before the data swap request was received.

[0174] For example, the network processor can process the third address information using a set of hash functions to obtain at least one hash value, i.e., the hash value corresponding to the third address information. The network processor can then obtain the write operation instruction number stored in at least one second target location indicated by the at least one hash value corresponding to the third address information from the current filter, thereby obtaining one or more latest operation identifiers.

[0175] S1203, determine whether the latest operation identifier is greater than the playback progress identifier; if yes, proceed to step S1204; if no, proceed to step S1205.

[0176] S1204, Ignore this data swapping request.

[0177] S1205, execute the data swapping request.

[0178] The network processor can determine whether to execute the data swap request based on the playback progress indicator and the latest operation indicator.

[0179] In one embodiment, when a latest operation identifier is obtained in step S1202, if the latest operation identifier is greater than the playback progress identifier S_MAX carried in the data swapping request, it means that the write-ahead operation log corresponding to the third address information is still stored in the storage space corresponding to the current filter and has not yet been played back or is being played back. The data A corresponding to the third address information is about to be modified or is being modified. In this case, the network processor ignores the data swapping request and does not perform data swapping to avoid swapping old data or part of the old data into the cache area of ​​the acceleration device.

[0180] When multiple latest operation identifiers are obtained in step S1202, if each latest operation identifier is greater than the playback progress identifier S_MAX carried in the data swap request, it can be determined that the latest operation identifier corresponding to the third address information is greater than the playback progress identifier S_MAX. This indicates that the data A corresponding to the third address information is about to be modified or is being modified. The network processor can ignore the data swap request and not perform the data swap. As mentioned above, since different address information may yield the same value after hashing, if any latest operation identifier is less than the playback progress identifier carried in the data swap request, then the latest operation identifier greater than the playback progress identifier may not be the write operation instruction number corresponding to the third address information, but may be the write operation instruction number corresponding to other address information. Therefore, based on the latest operation identifier less than the playback progress identifier, it can be assumed that there is no unplaybacked pre-write operation log corresponding to the third address information. The network processor can perform the data swap operation and save the data A stored in the data storage area to the cache area of ​​the acceleration device according to the third address information.

[0181] In another embodiment, in step S1202, if the network processor obtains multiple hash values ​​corresponding to the third address information, multiple second target locations can be determined in the current filter based on the multiple hash values. The network processor can obtain the write operation instruction number stored in each of the multiple second target locations, obtaining multiple write operation numbers, and using the smallest write operation number among the multiple write operation numbers as the latest operation identifier. If the latest operation identifier is less than the playback progress identifier, the data swapping request is executed; if the latest operation identifier is greater than the playback progress identifier, the data swapping request is ignored, and data swapping is not executed.

[0182] In the above embodiments, the network processor uses a monotonically increasing operation identifier to determine whether there is a latest write-ahead operation log for a certain address that has not yet been replayed by comparing the latest operation identifier with the playback progress identifier. If there is a latest write-ahead operation log that has not yet been replayed, the data swap-in request corresponding to the address is not executed, so as to avoid swapping the old data that is about to be modified into the cache area of ​​the acceleration device. For the same address information, the data stored in the cache area of ​​the acceleration device can be kept consistent with the data stored in the data storage area of ​​the storage device.

[0183] Based on the same technical concept as the above-described method embodiments, this application also provides a data storage device. This data storage device can be housed within the network processor of an acceleration device, which can be connected between a host and a storage device. In some embodiments, such as Figure 13 As shown, the data storage device 1300 may include a log generation unit 1301 and a log storage unit 1302. The data storage device 1300 can be used to implement the functions of the above method embodiments, and therefore can achieve the beneficial effects of the above method embodiments.

[0184] The log generation unit 1301 can be used to receive write operation instructions from the host forwarded by the interface in the acceleration device, and generate write-ahead operation logs based on the write operation instructions; the log storage unit 1302 can be used to save the write-ahead operation logs to the log storage area in the storage device; the write-ahead operation logs are used to save the data to be written carried in the write operation instructions to the data storage area in the storage device when the storage controller of the storage device is used for playback.

[0185] In some embodiments, the log storage unit 1302 can also be used to: send an operation execution completion notification to the host if it is determined that the write-ahead operation log has been saved.

[0186] In some embodiments, the log storage unit 1302 may be specifically used to: save a write-ahead operation log to a first storage space in the log storage area; the write-ahead operation log includes first address information carried in the write operation instruction; the first storage space is any one of at least one storage space; update a first filter corresponding to the first storage space according to the first address information in the write-ahead operation log; the updated first filter is used to characterize that the write-ahead operation log corresponding to the first address information is stored in the first storage space and awaits playback by the storage controller of the storage device.

[0187] In some embodiments, the first filter is stored in the acceleration device. The data storage device 1300 may further include a read instruction execution unit connected to the log storage unit 1302. The read instruction execution unit may be used to: receive a read operation instruction sent by the host, obtain the second address information carried in the read operation instruction; the read operation instruction is used to instruct data to be retrieved from the data storage area according to the second address information; if, based on the first filter, it is determined that the write-ahead operation log corresponding to the second address information is stored in the first storage space, then a delayed read notification is sent to the storage controller of the storage device, the delayed read notification being used to instruct the storage controller to wait for the write-ahead operation log corresponding to the second address information to be replayed before executing the read operation instruction.

[0188] In some embodiments, the log saving unit 1302 can also be used to: save the first filter to the storage device if it is determined that the first storage space is full.

[0189] In some embodiments, the read instruction execution unit may also be used to: if it is determined based on the first filter that the write-ahead operation log corresponding to the second address information is not stored in the first storage space, send a read operation instruction to the storage controller of the storage device, so that the storage controller determines based on the filter in the storage device whether the write-ahead operation log corresponding to the second address information is stored in the log storage area, and waits for playback by the storage controller of the storage device.

[0190] In some embodiments, the write-ahead operation log includes the number of the write operation instruction; the log storage unit 1302 can be specifically used to: determine at least one first target location in the first filter according to the first address information, and save the number of the write operation instruction to each of the at least one first target location.

[0191] In some embodiments, the write operation instructions are numbered sequentially according to the time order in which each write operation instruction is received by the network processor. The data storage device 1300 may further include a data swapping unit connected to a log storage unit. The data swapping unit can be used to: receive a data swapping request sent by the storage controller of the storage device; obtain the third address information and playback progress identifier carried in the data swapping request; the playback progress identifier refers to the maximum number contained in the pre-write operation log that the storage controller has completed playback of; the data swapping request is used to instruct that data stored in the data storage area be saved to the cache area of ​​the acceleration device according to the third address information; obtain the latest operation identifier from a first filter according to the third address information; the latest operation identifier refers to the number of the write operation instruction corresponding to the last received third address information before receiving the data swapping request; if the latest operation identifier is greater than the playback progress identifier, the data swapping request is ignored.

[0192] In some embodiments, the data swapping unit may be specifically used to: determine at least one second target location in a first filter based on third address information; obtain the number of the write operation instruction stored in each of the at least one second target location to obtain at least one write operation number; and take the smallest write operation number among the at least one write operation number as the latest operation identifier.

[0193] In some embodiments, the log storage unit 1302 may be specifically used to: save the write-ahead operation log to the main log storage area and the backup log storage area in the storage device, respectively.

[0194] and Figure 7 The illustrated method embodiments are based on the same technical concept. This application also provides a chip, which can be a computing chip and can be applied to the network processor of the above embodiments. This chip can be used to implement... Figure 7 The method embodiments shown can achieve the beneficial effects of the above-described method embodiments by providing the functionality of the methods described above.

[0195] In some embodiments, the structure of the chip 1400 can be as follows: Figure 14As shown, the device includes a processor 1401 and a power supply circuit 1402 connected to the processor 1401. The processor 1401 and the power supply circuit 1402 can be interconnected via a bus. The processor 1401 can be a digital signal processor (DSP), ASIC, field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other specific integrated circuits. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into address bus, data bus, control bus, etc. The power supply circuit 1402 is used to supply power to the processor 1401 via the bus.

[0196] The processor 1401 can be connected to a memory located outside the chip or to a memory located inside the chip, and run software programs and modules stored in the memory to perform various functional applications and data processing of the chip 1400, such as the data storage method provided in the embodiments of this application.

[0197] In some embodiments, the processor 1401 may include one or more processing units, which may be independent devices or integrated into one or more processors. The processor 1401 may also include a controller, which can generate operation control signals according to the instruction opcode and timing signals to control the instruction fetching and execution.

[0198] Based on the same technical concept as the above embodiments, this application also provides an acceleration device that can be connected between a host and a storage device. This acceleration device can be used to implement the functions of the above method embodiments, and therefore can achieve the beneficial effects of the above method embodiments.

[0199] In some embodiments, the structure of the acceleration device 1500 can be as follows: Figure 15 As shown, the interface includes an interface 1501 and a network processor 1502 connected to the interface 1501. The interface 1501 and the network processor 1502 can be interconnected via a bus. The interface 1501 can be a bus interface, data line interface, or other communication interface, used for communication with the host, receiving data sent by the host, and providing the received data to the network processor 1502. The network processor 1502 can be an NP that supports a programmable architecture and has remote direct data access capabilities. The network processor 1502 can employ... Figure 14 The chip shown. The bus can be a PCI bus or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0200] The network processor 1502 can be connected to a memory located outside the acceleration device or to a memory located inside the acceleration device, run software programs and modules stored in the memory, and interact with the host and the storage device. This allows it to execute the data storage method provided in the embodiments of this application.

[0201] In some embodiments, the network processor 1502 may include one or more processing units, which may be independent devices or integrated into one or more processors. The network processor 1502 may also include a controller, which can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0202] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the acceleration device. In other embodiments of this application, the acceleration device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0203] Based on the same technical concept as the above-described method embodiments, this application also provides a data storage system, the structure of which can be as follows: Figure 3 As shown, the system includes a host 610, an acceleration device 620, and a storage device 700. The acceleration device 620 is connected between the host 610 and the storage device 700. The acceleration device 620 may include a network processor 622, which can be used to execute computer programs to implement the functions of the above-described method embodiments.

[0204] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing a computer program or instructions. The computer program or instructions can constitute a computer program product.

[0205] This application also provides a computer program product comprising computer-executable instructions. In one embodiment, the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.

[0206] Computer-executable instructions can be stored in a computer-readable storage medium. This application also provides a computer-readable storage medium storing executable instructions. In one embodiment, the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.

[0207] The computer-readable storage medium provided in the embodiments of this application may be random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, portable hard disk, CD-ROM, or any other form of computer-readable storage medium known in the art.

[0208] Computer-executable instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive.

[0209] One or more of the above modules or units can be implemented by software, hardware, or a combination of both. When any of the above modules or units is implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: CPU, microprocessor, digital signal processor (DSP), microcontroller unit (MCU), or artificial intelligence processor, etc., various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC, DPU, or ASIC, or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as FPGAs, PLDs, or logic circuits that implement dedicated logic operations.

[0210] When the above modules or units are implemented in hardware, the hardware can be any one or any combination of CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, special purpose digital circuit, hardware accelerator or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data storage method, characterized in that, The method is applied to an acceleration device connected between a host and a storage device; the method includes: The interface in the acceleration device receives the write operation command sent by the host and forwards the write operation command to the network processor in the acceleration device. The network processor is used to perform the following operations: Generate a write-ahead log based on the write operation instruction; The write-ahead operation log is saved to the log storage area in the storage device; the write-ahead operation log is used to save the data to be written carried in the write operation instruction to the data storage area of ​​the storage device when it is played back through the storage controller of the storage device. If it is determined that the write-ahead operation log has been saved successfully, an operation completion notification is sent to the host.

2. The method according to claim 1, characterized in that, The method further includes: If the interface receives a non-modification operation instruction sent by the host, it sends the non-modification operation instruction to the main processor in the acceleration device or the storage controller of the storage device for processing; the non-modification operation instruction includes any operation instruction other than the write operation instruction.

3. The method according to claim 1 or 2, characterized in that, The log storage area includes at least one storage space; each of the at least one storage space is provided with a corresponding filter; the write-ahead operation log includes the first address information carried in the write operation instruction; Saving the write-ahead operation log to the log storage area in the storage device includes: The write-ahead operation log is saved to the first storage space, and the first filter corresponding to the first storage space is updated; the first storage space is any one of the at least one storage space; the updated first filter is used to indicate that the write-ahead operation log corresponding to the first address information is saved in the first storage space.

4. The method according to claim 3, characterized in that, The first filter is stored in the acceleration device, and the method further includes: Upon receiving a read operation instruction sent by the host, the second address information carried in the read operation instruction is obtained; If, based on the first filter, it is determined that the write-ahead operation log corresponding to the second address information is stored in the first storage space, a delayed read notification is sent to the storage controller of the storage device. The delayed read notification is used to instruct the storage controller to wait for the write-ahead operation log corresponding to the second address information to be replayed before executing the read operation instruction.

5. The method according to claim 4, characterized in that, The method further includes: If it is determined that the first storage space is full, then the first filter is saved to the storage device.

6. The method according to claim 4, characterized in that, The method further includes: If, based on the first filter, it is determined that the write-ahead operation log corresponding to the second address information is not stored in the first storage space, then the read operation instruction is sent to the storage controller of the storage device for processing.

7. The method according to claim 3, characterized in that, The write-ahead log includes the number of the write operation instruction; the number of the write operation instruction is an incrementing number according to the time sequence in which the network processor receives each write operation instruction. Updating the first filter corresponding to the first storage space includes: Based on the first address information, at least one first target location is determined in the first filter, and the number of the write operation instruction is saved to each of the at least one first target location.

8. The method according to claim 7, characterized in that, The method further includes: Upon receiving a data swap-in request from the storage controller of the storage device, the third address information and playback progress identifier carried in the data swap-in request are obtained; the playback progress identifier refers to the maximum number contained in the write-ahead operation log of the storage controller that has completed playback; the data swap-in request is used to instruct that the data stored in the data storage area be saved to the cache area of ​​the acceleration device according to the third address information. Obtain the latest operation identifier corresponding to the third address information from the first filter; the latest operation identifier refers to the number of the last write operation instruction corresponding to the third address information received before the data swapping request was received; If the latest operation identifier is greater than the playback progress identifier, then the data swapping request is ignored.

9. A data storage device, characterized in that, A network processor for use in an acceleration device connected between a host and a storage device; the apparatus includes: The log generation unit is used to receive the write operation instruction of the host forwarded by the interface in the acceleration device, and generate a pre-write operation log based on the write operation instruction. The log storage unit is used to save the write-ahead operation log to the log storage area in the storage device, and to send an operation execution completion notification to the host when it is determined that the write-ahead operation log has been saved; the write-ahead operation log is used to save the data to be written carried in the write operation instruction to the data storage area in the storage device when it is played back through the storage controller of the storage device.

10. The apparatus according to claim 9, characterized in that, The log storage area includes at least one storage space; each of the at least one storage space is correspondingly provided with a filter; the write-ahead operation log includes first address information carried in the write operation instruction; the log storage unit is specifically used for: The write-ahead operation log is saved to the first storage space, and the first filter corresponding to the first storage space is updated; the first storage space is any one of the at least one storage space. The updated first filter is used to indicate that the write-ahead operation log corresponding to the first address information is stored in the first storage space.

11. The apparatus according to claim 10, characterized in that, The first filter is stored in the acceleration device, and the device further includes: The read instruction execution unit is used to receive a read operation instruction sent by the host, obtain the second address information carried in the read operation instruction; if it is determined based on the first filter that the write-ahead operation log corresponding to the second address information is stored in the first storage space, then a delayed read notification is sent to the storage controller of the storage device. The delayed read notification is used to instruct the storage controller to wait for the write-ahead operation log corresponding to the second address information to be replayed before executing the read operation instruction.

12. A chip, characterized in that, It includes a processor and a power supply circuit; the power supply circuit is used to supply power to the processor, and the processor is used to execute a computer program to perform the method executed by the network processor as described in any one of claims 1 to 8 on data obtained through an interface.

13. An acceleration device, characterized in that, The acceleration device is connected between the host and the storage device, and the acceleration device includes an interface and a network processor. The interface is used to receive data and provide the received data to the network processor; The network processor is used to execute the method performed by the network processor according to any one of claims 1 to 8.

14. A data storage system, characterized in that, It includes a host, a storage device, and the acceleration device as described in claim 13.

Citation Information

Patent Citations

  • Data storage method and electronic equipment

    CN105787129A

  • Storage system, data processing method and device, storage system and electronic equipment

    CN113157628A

  • Metadata processing method in storage equipment and related equipment

    CN113805789A