Data processing method and device and medium

Abstract

The invention discloses a data processing method and device and a medium, and relates to the technical field of data processing. A hardware acceleration strategy or a software processing strategy is determined through flexible judgment according to different operation commands, and the diversity and flexibility of processing paths are improved. When the processing strategy is the hardware acceleration strategy, the hardware acceleration process can rapidly process the complex command, the calculation burden of the software controller is reduced, and after acceleration processing is conducted through the hardware acceleration unit, reconstruction processing is conducted on the complex command and the operation command, so that data transmission is conducted on the reconstructed data, and the reconstructed data are sent to the software controller. In the process, computing power resources of data carrying transmission and data processing between the host and the software controller are saved, and the response speed is increased in a hardware acceleration mode. For reconstruction processing, the software controller is considered to carry out subsequent access management processing, and the integrity of data processing is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method, device and medium. BACKGROUND

[0002] In a data storage processing process, a Non-Volatile Memory Express (NVME) controller processes commands and data processing methods in a single software manner, such as parsing MVME commands, driving data transmission and carrying operations. Due to the complexity of the software processing flow, and the heavy load of the Central Processing Unit (CPU) or firmware in the software processing process, the processing speed is slow, and the CPU or firmware processing performance under the software processing manner is reduced.

[0003] Therefore, how to improve the data storage performance and efficiency is an urgent problem to be solved by those skilled in the art. SUMMARY

[0004] The purpose of the present application is to provide a data processing method, device and medium to solve the problem that the conventional storage processing adopts a single software manner, resulting in a complex processing flow, heavy load, slow speed and reduced processing performance.

[0005] To solve the above technical problems, the present application provides a data processing method, comprising:

[0006] responding to an operation command issued by a host; and determining a corresponding processing strategy based on the type of the operation command;

[0007] when the processing strategy is a hardware acceleration strategy, performing acceleration processing on the operation command through a hardware acceleration unit to obtain first processing information; and after reconstructing the first processing information and the operation command, sending them to the software controller for access processing by the software controller;

[0008] when the processing strategy is a software processing strategy, sending the operation command to the software controller for data processing.

[0009] On the one hand, the corresponding processing strategy is determined based on the type of the operation command, comprising:

[0010] determining the command type and command flag of the operation command, wherein the command type includes a data operation command and a management command;

[0011] obtaining the queue depth and the preset queue depth of a storage device;

[0012] When the command type is a data operation command, the command flag is a forced unit access flag, and the queue depth is greater than the preset queue depth, the processing strategy is determined as a hardware acceleration strategy.

[0013] On the other hand, a corresponding processing strategy is determined based on the type of the operation command, including:

[0014] A command type of the operation command is determined, wherein the command type includes a data operation command and a management command.

[0015] A current load of the software controller is obtained.

[0016] Bandwidth data corresponding to a data transmission channel between the storage device and the host is obtained.

[0017] When the current load exceeds a preset load, the bandwidth data is less than residual bandwidth data, and the command type is a data operation command, the processing strategy is determined as a hardware acceleration strategy.

[0018] On the other hand, the operation command is processed by a hardware acceleration unit to obtain first processing information, including:

[0019] The hardware acceleration unit is controlled to receive the operation command and analyze address information of the operation command to perform memory address allocation processing to obtain corresponding first address information.

[0020] A lock register address is obtained, and a lock state is read.

[0021] When the lock state is an idle state, the lock state corresponding lock is occupied to complete locking processing of the operation command.

[0022] The operation command after the locking processing is processed according to the first address information to obtain the first processing information.

[0023] On the other hand, a determination process of the lock register address includes:

[0024] The host reads a start address of the hardware acceleration unit through an interface configuration space to determine a base address of a memory mapping region of the operation command.

[0025] In the memory mapping region, a lock register is allocated according to the base address.

[0026] Address information of the lock register of the hardware acceleration unit is mapped to a corresponding addressing space to obtain the lock register address.

[0027] On the other hand, the hardware acceleration unit is controlled to perform memory address allocation processing on the operation command to obtain corresponding first address information, including:

[0028] controlling the hardware acceleration unit to apply a physical region page to a host according to data information and command information of the operation command;

[0029] allocating a memory address space of the corresponding hardware acceleration unit for the physical region page;

[0030] controlling the hardware acceleration unit to distribute a list of the physical region page through a direct memory access engine and embed the list into the operation command;

[0031] configuring a source address and a target address corresponding to the operation command as the first address information according to the embedded operation command through the direct memory access engine of the hardware acceleration unit.

[0032] On the other hand, the first processing information and the operation command are reconstructed, including:

[0033] obtaining a lock address and a lock serial number corresponding to the current operation command;

[0034] obtaining the first address information allocated by the current operation command;

[0035] establishing a mapping relationship between the current operation command, the lock address, the lock serial number and the first address information to determine first reconstruction information;

[0036] obtaining a lock property corresponding to the previous operation command and a lock property corresponding to the current operation command;

[0037] if the lock properties corresponding to the previous operation command and the current operation command are synchronous access respectively, then the first reconstruction information corresponding to the previous operation command and the current operation command is pre-processed to obtain first pre-reconstruction information;

[0038] on the basis of the first pre-reconstruction information, judging whether the first address information corresponding to the previous operation command and the current operation command respectively exists same address information;

[0039] if there is same address information, then the first reconstruction information corresponding to the current operation command is in a different address chain table with the previous operation command, and is reconstructed with other operation commands;

[0040] if there is no same address information, then the data amount corresponding to the current operation command is added to the first pre-reconstruction information to complete the reconstruction processing of the current operation command and the previous operation command.

[0041] On the other hand, after the first processing information and the operation command are reconstructed, they are sent to the software controller, including:

[0042] Obtaining target information corresponding to the operation command after reconstruction processing;

[0043] Reserving operation information corresponding to the software controller in the target information for sending to the software controller;

[0044] Deleting operation information not corresponding to the software controller in the target information to release the corresponding memory space.

[0045] To solve the above technical problems, the present application also provides a data processing device, comprising:

[0046] A memory for storing a computer program;

[0047] A processor for executing the computer program to implement the steps of the data processing method.

[0048] To solve the above technical problems, the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to implement the steps of the data processing method.

[0049] The beneficial effects of the present application are as follows: first, based on the type of operation command issued by the response host, different processing strategies are determined. Compared with the conventional single software processing method using software controllers (such as CPU or firmware), the present application determines hardware acceleration strategy or software processing strategy for different operation commands, improves the diversity of processing path, and also improves the flexibility of data processing. Second, when the processing strategy is hardware acceleration strategy, the hardware acceleration flow can quickly process complex commands, reducing the computational burden of the software controller, also reducing the frequency of software accessing hardware, improving the overall processing speed. At the same time, it also shares part of the command processing work, so that the resources of software and hardware are reasonably allocated, and the computing power is balanced. Third, during the hardware acceleration strategy processing, after acceleration processing by the hardware acceleration unit, reconstruction processing is performed on the operation command, so as to transmit the reconstructed data to the software controller. In this process, the data transfer and data processing computing resources between the host and the software controller are saved, and the response speed is improved through hardware acceleration. As for the reconstruction processing, it is considered that the software controller performs subsequent access management processing to ensure the integrity of data processing.

[0050] When the command and the corresponding queue depth meet the respective conditions, the processing strategy is determined as the hardware acceleration strategy, based on the type of the NVME command and the characteristics of the storage device, the accuracy of the judgment is improved while the system processing performance is optimized. Based on the type of the operation command, the corresponding processing strategy is determined, and through the consideration of three different factors, the NVME command is quickly responded, the delay of command processing is reduced, and the response speed of the system is improved. Part of the task is transferred from the CPU to the hardware acceleration unit, which can reduce the usage rate of the CPU and thus reduce the power consumption of the system. The hardware acceleration unit is used to perform the acceleration process of the locking processing and the memory address allocation processing, and the hardware acceleration unit usually has special hardware resources, such as a dedicated processor core or hardware logic, which can execute the locking and memory address allocation operations faster, thereby improving the overall system performance. The locking and memory address allocation processing is transferred from the CPU to the hardware acceleration unit, which can reduce the burden of the CPU and enable it to focus on other computationally intensive tasks. The lock mechanism is realized through the memory mapping of the hardware acceleration unit, which can be acquired or released by accessing a specific memory address, effectively manages the access to the NVME command, ensures the consistency and safety of the data, and improves the command processing efficiency by utilizing the high-performance characteristics of the hardware acceleration unit.

[0051] In addition, the application also provides a data processing device and a medium, which have the same beneficial effects as the data processing method described above. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application, the drawings required in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 The flowchart of the conventional software processing mode;

[0054] Figure 2 The flowchart of the data processing method provided by the embodiment of the present application;

[0055] Figure 3 The schematic diagram of the reconstruction processing provided by the embodiment of the present application;

[0056] Figure 4 The structural schematic diagram of the data processing device provided by the embodiment of the present application;

[0057] Figure 5 The flowchart of another data processing method provided by the embodiment of the present application;

[0058] Figure 6A structural diagram of a data transmission device provided for an embodiment of the present application is shown in FIG. 1.

[0059] Figure 7 A structural diagram of a data transmission device provided for an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0061] The core of the present application is to provide a data processing method and device and medium, so as to solve the problem that the conventional storage processing adopts a single software processing mode, resulting in a complex processing flow, heavy load, slow speed and reduced processing performance.

[0062] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0063] In some existing NVME controllers, all commands are uniformly processed by software. The NVME controller obtains NVME commands from the host end, and directly delivers the commands to software (i.e. CPU or Firmware). The software senses the new commands and performs corresponding processing. If data transfer between the host and the NVME device is required, the software drives the data transfer module to perform data transfer. After the command execution is completed, the software is responsible for replying the command completion information to the host end.

[0064] CPU or Firmware undertakes heavy tasks, not only to parse commands, but also to drive data transmission and handle a series of operations such as completion status. In the process of command parsing, the work of CPU or Firmware is extremely complex. For example, interacting with hardware, when transmitting commands, hardware resources need to be accurately controlled to ensure that the commands can be accurately executed. In order to guarantee the atomic operation between commands and avoid data confusion, lock processing is also set. At the same time, the address of the data in the command written to the NVME device storage space also needs to be finely managed and allocated, and a series of operations need to consume a lot of computing resources and time. In the process of data transmission, CPU or Firmware also needs to coordinate the processing of multiple links. It needs to frequently access hardware resources, control data transfer, and monitor the transfer state in real time. Any error in any link may cause data transmission failure or error. In addition, the interaction process between software and hardware needs to access system memory or hardware units through the bus in the NVME device, and these behaviors will also consume a certain amount of time. This single software processing method has been difficult to meet the requirements of efficient data processing, and has seriously affected the performance and response speed of the NVME controller.

[0065] Figure 1 The flowchart of the conventional software processing method is shown in FIG. 1, and the steps include: Figure 1

[0066] S11: The host exists a command;

[0067] S12: The command is obtained;

[0068] S13: The command is transmitted to the software;

[0069] S14: It is judged whether data transfer is needed, if yes, step S15 is entered, if not, step S16 is entered;

[0070] S15: The software drives the data transmission module to perform data transmission processing;

[0071] S16: The software executes the command;

[0072] S17: The software replies completion.

[0073] ​In the full-software command processing mode currently adopted by the NVME controller, the CPU or Firmware bears excessive work load. They not only need to analyze the obtained NVME command, but also are responsible for driving data transmission and processing completion status and a series of operations, which greatly increases the calculation amount of the software. In processing a large number of complex commands, the processing speed of the CPU or Firmware will be significantly reduced. Because complex commands often involve more calculation steps and resource allocation, and the processing capacity of the CPU or Firmware is limited, when the task amount exceeds its load, performance bottleneck will occur, resulting in reduced command processing efficiency. The data processing method provided by the present application can solve the above technical problems.

[0074] Figure 2 A flowchart of a data processing method provided for an embodiment of the present application is shown in Figure 2 The method comprises:

[0075] S21: responding to the operation command issued by the host; and determining the corresponding processing strategy based on the type of the operation command;

[0076] S22: determining whether the processing strategy is a hardware acceleration strategy, if yes, proceeding to step S23, if not, proceeding to step S24;

[0077] S23: when the processing strategy is a hardware acceleration strategy, performing acceleration processing on the operation command through a hardware acceleration unit to obtain first processing information; and after reconstructing the first processing information and the operation command, sending them to a software controller for access processing by the software controller;

[0078] S24: when the processing strategy is a software processing strategy, sending the operation command to the software controller for data processing.

[0079] Specifically, responding to the operation command issued by the host can be an NVME operation command or other operation command, which is not limited here, and the corresponding processing strategy is determined based on the type of the command. The determination process here can be based only on the type of the command, or other parameter factors can be added based on the type of the command to determine. As for the command type, it is determined whether the command needs to be processed by hardware acceleration, such as complex commands involving a large amount of data read-write operation, which can be processed by hardware acceleration. Some simple query commands can be selected not to be processed by hardware acceleration, but directly processed by software. In addition, the processing strategy here can be in a constantly open state or a long closed state, and the opening is performed when the processing strategy is determined, which is not limited here.

[0080] As to the command type, it can be a regular data operation command requiring performance acceleration, such as read and write operation commands, or a management command not requiring performance acceleration, such as a command mainly used for device configuration and state query, which can be directly classified and processed to determine the corresponding processing strategy. In addition, in addition to the command type setting, it can also be determined whether hardware acceleration processing is required based on the current software controller processing and corresponding data transmission. For example, if the current software controller has a large load corresponding to the processing, or occupies a large bandwidth resource in the transmission process, the operation command will further increase the current software controller or bandwidth in the transmission process, and the hardware controller using hardware acceleration mode can be used for processing.

[0081] When it is determined to be a hardware acceleration strategy, hardware acceleration unit processing is performed. The hardware acceleration unit here is not limited to a field programmable gate array (FPGA) or an acceleration card, but can also use a combination of editable combination logic, such as NAND gate, etc. It is not limited here and can be set according to the actual situation.

[0082] The operation command is accelerated by the hardware acceleration unit to obtain first processing information. The acceleration processing operation here is different from the processing operation corresponding to the regular software controller. Here, the hardware response mechanism of the hardware acceleration unit is combined to improve the response mechanism and also improve the data processing efficiency. For example, lock processing and memory address allocation processing, etc. can be the same as the regular hardware acceleration processing mode, or a new acceleration processing mode can be set based on a new hardware accelerator or combination logic, etc. It is not limited here.

[0083] The first processing information is reconstructed with the operation command. Here, the reconstruction is to combine the information under the address or lock of the above acceleration processing process with the operation command, so as to perform the regular subsequent processing operation after sending to the software controller. It should be noted that in the embodiment, the acquisition and transmission process of the NVME command between the host and the storage device can be realized through the NVME controller or the CPU. However, in order to alleviate the load capacity of the CPU, the processing can be performed through the NVME controller, but in the subsequent data access processing of the CPU, it is still attributed to the CPU processing, so there will be a reconstruction process.

[0084] After the reconstruction processing, it is sent to the software controller. It can be all data transmission, or the data of the reconstruction processing can be simplified to leave only the necessary information according to the subsequent access processing required by the software controller.

[0085] The software processing strategy in step S24 is different from a conventional CPU processing manner or a firmware processing manner.

[0086] The embodiment of the present application has the following beneficial effects. First, different processing strategies are determined based on the type of operation command issued by the host. Compared with the conventional single software processing manner using a software controller (such as a CPU or firmware), the present application determines a hardware acceleration strategy or a software processing strategy for different operation commands, thereby improving the diversity of processing paths and the flexibility of data processing. Second, when the processing strategy is a hardware acceleration strategy, the hardware acceleration flow can quickly process complex commands, thereby reducing the computational burden of the software controller, reducing the frequency of software accessing hardware, and improving the overall processing speed. At the same time, the hardware acceleration strategy also shares part of the command processing work, so that the resources of software and hardware are reasonably allocated, and the computing power is balanced. Third, during the hardware acceleration strategy processing, the operation command is reconstructed after being processed by the hardware acceleration unit, so as to transmit the reconstructed data to the software controller. In this process, the data transfer and data processing computing resources between the host and the software controller are saved, and the response speed is improved through the hardware acceleration manner. The reconstruction processing is considered for the subsequent access management processing of the software controller, thereby ensuring the integrity of data processing.

[0087] In some embodiments, the corresponding processing strategy is determined based on the type of operation command, including:

[0088] The command type and the command flag of the operation command are determined, wherein the command type includes a data operation command and a management command.

[0089] The queue depth of the storage device and the preset queue depth are obtained.

[0090] When the command type is the data operation command, the command flag is the forced unit access flag, and the queue depth is greater than the preset queue depth, the processing strategy is determined to be the hardware acceleration strategy.

[0091] Specifically, data operation commands in the command type, which need performance acceleration, directly involve read and write operations of data. Management commands (such as Identify, Get Log Page, etc.) generally do not need performance acceleration, because they are mainly used for configuration and state query of the device. The queue depth of the storage device, which can be a hard disk or other devices, etc. NVME supports multiple deep queues, each of which can support up to 64K commands. If the queue depth is deep (for example, exceeds a certain threshold), it can be considered that the Input / Output (I / O) commands in the queue need performance acceleration to fully utilize the parallel processing capability of the multi-core processor. The queue depth is the number of commands that can be accommodated in the submission queue of the NVME device at the same time. The higher the queue depth, the more commands the device can handle at the same time, thereby improving concurrency. By increasing the queue depth, the time for the host to wait for the device to process the command can be reduced, thereby reducing the average delay. Regarding the setting of the queue depth, the hardware design of the NVME device determines the maximum queue depth it supports. For example, some high-end NVME solid state drives (SSD) can support a queue depth of up to 64K, while some entry-level devices can only support a smaller queue depth.

[0092] Command flags, if a write command has a Force Unit Access (FUA) flag, it means that the data needs to be written directly to the non-volatile storage medium, rather than first written to the cache, in which case the command may need performance acceleration to ensure that the data is quickly written and reduce latency. The Force Unit Access flag indicates that the storage device must directly access the storage unit (such as the physical sector on the hard disk drive), rather than using any cache mechanism. Enabling FUA can ensure that the data of the write operation is directly written to the permanent storage medium, rather than first written to the cache, which can reduce the risk of data loss and improve data persistence.

[0093] When the command type is a data operation command and the command flag is a Force Unit Access flag, and the queue depth is greater than the preset queue depth, it is listed as a hardware acceleration strategy.

[0094] The embodiment provided in this embodiment determines the processing strategy as a hardware acceleration strategy when the command and the corresponding queue depth meet the respective conditions, based on the type of NVME command and the characteristics of the storage device, to improve the accuracy of the judgment while optimizing the system processing performance.

[0095] In other embodiments, the corresponding processing strategy is determined based on the type of operation command, including:

[0096] determining a command type of the operation command, wherein the command type comprises a data operation command and a management command;

[0097] obtaining a current load of the software controller;

[0098] obtaining bandwidth data corresponding to a data transmission channel between the storage device and the host;

[0099] when the current load exceeds the preset load, the bandwidth data is less than the residual bandwidth data, and the command type is the data operation command, determining that the processing strategy is the hardware acceleration strategy.

[0100] Specifically, in combination with the above-mentioned command type, considering the current load of the software controller and the bandwidth data corresponding to the data transmission channel, the processing strategy is determined under sufficient judgment of the three parameter factors.

[0101] In the software processing process, according to the current load of the software controller and the bandwidth data existing in the data transmission channel corresponding to the host and the storage device, if the current load exceeds the preset load and the bandwidth data is less than the residual bandwidth data, it indicates that the load condition of the software controller is relatively tense, and the bandwidth resource reserved for the subsequent command corresponding to the bandwidth data is less, so the processing path of hardware acceleration can be used for processing. When the CPU load exceeds the preset load, the CPU can not be able to efficiently process a large number of I / O requests, resulting in a decrease in system performance. By using a hardware acceleration unit (such as FPGA or a dedicated acceleration card) to process NVME commands, part of the computing task can be transferred from the CPU to the hardware acceleration unit, thereby reducing the burden of the CPU and improving the overall performance of the system. If the bandwidth of data transmission is less than the residual bandwidth, it indicates that the current CPU or system bus can not fully utilize the high performance of the NVME device. The hardware acceleration unit can utilize its dedicated direct memory access (Direct Memory Access, DMA) channel and optimized data transmission path to directly transmit data from the host memory to the NVME device, thereby improving the data transmission efficiency and fully utilizing the high bandwidth characteristics of the NVME device. The hardware acceleration unit usually has an optimized read-write path and cache mechanism, which can quickly process NVME commands and improve the performance of read-write operations.

[0102] The embodiment provided determines the corresponding processing strategy based on the type of operation command, considers three different factors to quickly respond to NVME commands, reduces the delay of command processing, and thereby improves the response speed of the system. Transferring part of the task from the CPU to the hardware acceleration unit can reduce the usage rate of the CPU, thereby reducing the power consumption of the system.

[0103] In some embodiments, the operation command is accelerated and processed by the hardware acceleration unit to obtain first processing information, including:

[0104] The control hardware acceleration unit receives an operation command and parses address information of the operation command to perform memory address allocation processing to obtain corresponding first address information;

[0105] The lock register address is obtained and the lock state is read;

[0106] If the lock state is an idle state, the lock corresponding to the lock state is occupied to complete the locking processing of the operation command;

[0107] The operation command after the locking processing is processed according to the first address information to obtain first processing information.

[0108] Specifically, memory address allocation usually occurs before locking processing, because locking processing needs a specific memory address to operate the lock. Here, the memory requirement needs to be determined by the application program or the operating system to determine the memory size and type that needs to be allocated, and if the memory allocation is for a hardware device, the specific requirements of the device also need to be considered.

[0109] The hardware acceleration unit receives an operation command sent by the host, parses the address information in the command, determines the storage space address (first address information) where the data will be written, and allocates storage space in the storage medium of the NVME device for storing the data to be written.

[0110] The hardware acceleration unit reads the lock state by accessing the address of the lock register. If the lock is not occupied (the lock state is 0), the hardware acceleration unit sets the lock state to 1, indicating that the lock is occupied; if the lock is occupied (the lock state is 1), the hardware acceleration unit needs to wait until the lock is released.

[0111] The operation command after the locking processing is processed according to the first address information to obtain first processing information.

[0112] The acceleration process provided by the embodiment for locking processing and memory address allocation processing by the hardware acceleration unit usually has special hardware resources, such as dedicated processor cores or hardware logic, which can perform locking and memory address allocation operations faster, thereby improving the overall system performance. Transferring the locking and memory address allocation processing from the CPU to the hardware acceleration unit can reduce the burden of the CPU, so that it can focus on other computationally intensive tasks.

[0113] In some embodiments, the determination process of the lock register address includes:

[0114] The host reads the starting address of the hardware acceleration unit through the interface configuration space to determine the base address of the memory mapping region of the operation command;

[0115] In the memory mapping region, the lock register is allocated according to the base address;

[0116] The address information of the lock register of the hardware acceleration unit is mapped into the corresponding address space to obtain the lock register address.

[0117] Specifically, the hardware acceleration unit connects the NVME device through a Peripheral Component Interconnect Express (PCIE) interface, and the host communicates with the device through a standard NVME driver. When the hardware acceleration unit is initialized, a base address register (BAR) is configured. In the memory mapping region of the hardware acceleration unit, one or more registers are allocated as lock registers. The address space of the NVME controller is configured, and the address information of the lock register is mapped into the address space of the NVME controller. The address information of the lock register is read into the corresponding address space by accessing the address information of the lock register to obtain the address of the lock register. Each PCIE device can have multiple BARs, and each BAR corresponds to a memory mapping region. These regions are usually used for device register access, DMA operation, or internal storage space of the device.

[0118] The lock mechanism provided by the embodiment is realized through the memory mapping of the hardware acceleration unit. The lock can be obtained or released by accessing a specific memory address, the access to the NVME command is effectively managed, the data consistency and safety are ensured, and the command processing efficiency is improved by using the high performance characteristics of the hardware acceleration unit.

[0119] In some embodiments, the control hardware acceleration unit performs memory address allocation processing on the operation command to obtain corresponding first address information, including:

[0120] The control hardware acceleration unit applies a physical region page to the host according to the data information and the command information of the operation command;

[0121] A corresponding memory address space of the hardware acceleration unit is allocated for the physical region page;

[0122] The control hardware acceleration unit distributes a list of the physical region page through a direct memory access engine, and embeds the list into the operation command;

[0123] The control hardware acceleration unit configures the source address and the target address corresponding to the operation command as the first address information according to the embedded operation command through the direct memory access engine of the hardware acceleration unit.

[0124] Specifically, the hardware acceleration unit applies a physical region page (PRP) to the host according to data information of the operation command and cached command information, and calculates a host address of the command information. A corresponding memory address space (DMA-able memory) of the hardware acceleration unit is allocated to the PRP, ensuring the efficiency of data transmission.

[0125] A PRP list is constructed, and a data buffer related to the command is processed according to a requirement of the NVME command. If the data transmission is greater than a certain length, the PRP list needs to be constructed, which describes the physical address of the data in the memory. The hardware acceleration unit allocates the PRP list through a DMA engine and embeds it into the NVME command.

[0126] The DMA engine of the hardware acceleration card configures a source address and a target address of data transmission as first address information according to a requirement of the NVME command. The DMA engine transmits data from the host memory to the NVME device or from the NVME device to the host memory through a PCIE bus.

[0127] The hardware acceleration card provided in the embodiment efficiently allocates a memory address for the NVME command through the DMA engine and accelerates the data transmission process, thereby improving the overall performance of the system.

[0128] In some embodiments, the first processing information is reconstructed with the operation command, including:

[0129] An lock address corresponding to the current operation command and an lock sequence number are obtained;

[0130] First address information allocated for the current operation command is obtained;

[0131] The current operation command, the lock address, the lock sequence number, and the first address information are mapped to determine first reconstruction information;

[0132] An lock property corresponding to a previous operation command and an lock property corresponding to the current operation command are obtained;

[0133] If the lock properties corresponding to the previous operation command and the current operation command are synchronous access, respectively, first reconstruction information corresponding to the previous operation command and the current operation command is pre-processed to obtain first pre-reconstruction information;

[0134] On the basis of the first pre-reconstruction information, it is determined whether the first address information corresponding to the previous operation command and the current operation command has same address information;

[0135] If there is same address information, the first reconstruction information corresponding to the current operation command is in a different address chain table of the previous operation command, and is reconstructed with other operation commands.

[0136] If no identical address information exists, the data volume corresponding to the current operation command is added to the first pre-reconstruction information to complete the reconstruction process between the current operation command and the previous operation command.

[0137] Figure 3 This is a schematic diagram of a reconstruction process provided in an embodiment of the present invention, such as... Figure 3 As shown, the lock address and lock sequence number corresponding to the current operation command are needed. A mapping relationship is established between the current operation command, lock address, lock sequence number, and the allocated first address information to determine the first reconstruction information. Based on the acquired lock identification (ID), a memory address is allocated. This may involve looking up or calculating the memory address for subsequent data transfer. The command sending module receives the reconstructed command and prepares to send it. This module generates partial commands containing memory address information to guide data transfer. The data transfer module is responsible for handling the actual data transfer. It receives partial commands from the command sending module, which contain data volume information. Based on these commands, the data transfer module transfers data from the source address to the destination address. The commands generated by the command sending module contain a linked list of memory addresses, which may be a list of pointers to data buffers used for DMA operations.

[0138] The above only describes the reconstruction of the current operation command. However, during the reconstruction of multiple operation commands, it is necessary to check the lock properties. In the case of a mutex lock, it ensures that only one thread or process can access the shared resource at a time. A mutex lock restricts multiple commands from accessing the same resource simultaneously, thus reducing concurrency. If multiple commands need to access the same resource, the mutex lock will cause the commands to wait, increasing processing latency. The existence of mutex locks complicates command reconstruction because it is necessary to ensure that the mutual exclusion property of the lock is not violated during the reconstruction process. Therefore, it is necessary to consider whether the lock properties of the previous and current operation commands are mutex locks. If not, they are synchronous access locks, and we can temporarily perform pre-reconstruction processing on the two NVME commands to obtain the first pre-reconstruction information.

[0139] Based on this, it is determined whether the first address information corresponding to the two NVME commands has the same address information. If it does, it means that they cannot be reconstructed in the same address list. The two commands need to be separated, that is, reconstructed in a different address list from the previous operation command, and with other operation commands.

[0140] If no identical address information exists, the data corresponding to the current operation command is added to the first pre-reconstruction information to complete the reconstruction process of the two commands.

[0141] The reconstruction processing provided by the embodiment is to complete the reconstruction information of the operation command and the reconstruction of other operation commands, so as to send the address chain table to the software controller for subsequent operation processing.

[0142] In some embodiments, after the first processing information is reconstructed with the operation command, the reconstructed operation command is sent to the software controller, including:

[0143] Obtaining target information corresponding to the reconstructed operation command;

[0144] Reserving operation information corresponding to the software controller in the target information for sending to the software controller;

[0145] Deleting operation information not corresponding to the software controller in the target information to release the corresponding memory space.

[0146] Specifically, while sending to the software controller, the necessary information of the command needs to be reserved, which is the operation information corresponding to the software controller, transmitted to the CPU or firmware accessible location for CPU / firmware management or subsequent operation. The unnecessary operation information is deleted to release the storage space.

[0147] In the command reconstruction process provided by the embodiment, the necessary information of the command is sent to the software controller for subsequent access processing by the software controller, which also shares the load processing process of the software controller and saves the memory space of the software controller. In addition, deleting the unnecessary information also saves the memory space of the NVME controller.

[0148] In some embodiments, the processing strategy is determined based on the type of the operation command, including:

[0149] Obtaining a test command set and a test category of the operation command; wherein the test category at least includes performance processing speed, performance processing bandwidth and memory occupation;

[0150] According to the test category, the test command set is respectively tested by a hardware acceleration strategy and a software processing strategy to obtain a corresponding test result;

[0151] According to the test result and the corresponding test category, a final processing result is determined;

[0152] According to the processing result and the type of the operation command, a corresponding processing strategy is determined.

[0153] Specifically, the test is performed in two processing strategies in advance by using a test command set mode, and the test results are viewed in different test categories corresponding to the test, and the processing time, data transmission rate and memory usage of the hardware acceleration strategy and the software processing strategy are compared to determine the computing resource conditions of the two strategies.

[0154] Different test commands correspond to different types and may perform differently in computing resources, and therefore, the appropriate processing strategy needs to be selected according to the performance test results and the command type.

[0155] The pre-test for selecting the appropriate processing strategy provided by the embodiment significantly improves the performance of the storage device operation, reasonably allocates hardware and software resources according to the command type and the performance test results, and improves the overall efficiency of the system. Through the performance test and the dynamic selection strategy, the stability and reliability of the system are ensured, and the problems caused by resource contention or performance bottleneck are reduced.

[0156] Figure 4 A structural schematic diagram of a data processing device provided by an embodiment of the present application is shown in Figure 4 The command acquisition module, the command analysis module, the hardware acceleration module, the command reconstruction module, the command sending module, the data transmission module and the command completion module. The command acquisition module captures the NVME command from the host, and then delivers it to the command analysis module for analysis of each command. For admin commands and some I / O commands that do not require performance acceleration, such as copy commands, basic non-hardware acceleration processing can be performed; for some I / O commands that require processing performance, they are allocated to the hardware acceleration path for processing.

[0157] When processing a command that does not require hardware acceleration, the command parsing module delivers the command to the command sending module in a specified format (for example, adding a Submission Queue (SQ) ID, a Completion Queue (CQ) ID, a function ID, and the like to the original command), and the command sending module sends the command to the CPU or the Firmware. The command sending module can write the command to a memory space accessible by the CPU or the Firmware, such as a Static Random Access Memory (SRAM), so that the CPU or the Firmware reads the command. After the CPU or the Firmware obtains the command, for a command that does not require data transfer, the CPU or the Firmware performs a corresponding command operation, and then drives the command completion module to reply a completion status to the host; for a command that requires data transfer, the CPU or the Firmware determines a device memory address and performs lock management, and then drives the data transfer module to transfer data from the specified address to a specified location. After the data transfer module completes the data transfer, the data transfer module informs the CPU or the Firmware of completion information and error status, and the like. If there is a transfer error, the CPU or the Firmware performs an error processing procedure, such as retransmission, and if there is an error, the command completion module is driven to reply an error status to the host. If there is no error in the data transfer, the command completion module is directly driven to reply a completion status to the host.

[0158] When processing a command that requires hardware acceleration, the command parsing module drives the hardware acceleration module to process the command, and the hardware acceleration module performs operations such as Logical Block Addressing (LBA) locking and device memory address allocation. The command reconstruction module combines and reconstructs hardware acceleration information (such as a lock ID and device memory address information) and the original command, and then delivers the reconstructed command to the data transfer module. The data transfer module performs data transfer, and after the data transfer is completed, the data transfer module informs the command completion module to reply a completion status to the host. The command reconstruction module delivers the simplified command information to the command sending module (for example, deleting a host data address and the like that only need to be used by the data transfer module), and the command sending module sends the command to the CPU or the Firmware for subsequent management. If there is a data transfer error, the data transfer module also informs the command completion module of an error status, and the command completion module replies error information to the host.

[0159] Figure 5 A flowchart of another data processing method provided by an embodiment of the present application is shown in FIG. 4, and includes the following steps. Figure 5

[0160] S11: The host exists a command.

[0161] S12: The command is obtained.

[0162] ​S31: Analyze whether hardware acceleration, if yes, enter step S32, if no, enter step S13;

[0163] S32: Hardware acceleration operation;

[0164] S33: Command reconstruction;

[0165] S34: Data transfer;

[0166] S35: Hardware reply completion and feedback software;

[0167] S13: Command transmission to software;

[0168] S14: Determine whether data transfer is needed, if yes, enter step S15, if no, enter step S16;

[0169] S15: Software drive data transmission module for data transmission processing;

[0170] S16: Software executes command;

[0171] S17: Software reply completion.

[0172] The above detailed description of the data transmission method corresponds to each embodiment, on the basis of which, the application further discloses a data transmission device corresponding to the above method, Figure 6 A structural diagram of a data transmission device provided by the embodiment of the application is shown in FIG. 1. Figure 6 As shown in the figure, the data transmission device comprises:

[0173] A determination module 11 for responding to an operation command issued by a host; and determining a corresponding processing strategy based on the type of the operation command;

[0174] A first processing module 12 for, when the processing strategy is a hardware acceleration strategy, performing acceleration processing on the operation command through a hardware acceleration unit to obtain first processing information; and after reconstruction processing of the first processing information and the operation command, sending the first processing information and the operation command to a software controller for access processing by the software controller;

[0175] A second processing module 13 for, when the processing strategy is a software processing strategy, sending the operation command to the software controller for data processing.

[0176] Since the device part embodiments correspond to the above embodiments, the device part embodiments are described with reference to the above method part embodiments, which will not be described here.

[0177] For the data transmission device provided by the application, please refer to the above method embodiments, which will not be described here, and has the same beneficial effects as the above data transmission method.

[0178] Figure 7 A structural diagram of a data transmission device provided by an embodiment of the present application is shown in FIG. 1, which comprises: Figure 7

[0179] a memory 21 for storing a computer program;

[0180] a processor 22 for executing the computer program to implement the steps of the data transmission method.

[0181] The data transmission device provided by the embodiment can include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0182] The processor 22 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 can be implemented in at least one of a hardware form of a Digital Signal Processor (DSP), an FPGA, and a programmable logic array. The processor 22 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU. The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 22 can be integrated with a Graphics Processing Unit (GPU) for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 22 can further include an Artificial Intelligence (AI) processor for processing machine learning-related computing operations.

[0183] The memory 21 can include one or more computer-readable storage media, which can be non-transitory. The memory 21 can further include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In the embodiment, the memory 21 is at least used to store the following computer program 211, wherein the computer program is loaded and executed by the processor 22, and can implement the related steps of the data transmission method disclosed in any of the preceding embodiments. In addition, the resources stored by the memory 21 can further include an operating system 212 and data 213, etc., and the storage mode can be temporary storage or permanent storage. The operating system 212 can include Windows, Unix, Linux, etc. The data 213 can include, but is not limited to, data related to the data transmission method, etc.

[0184] In some embodiments, the data transmission device can further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.​

[0185] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the data transmission device, which can include more or fewer components than those shown in the drawings. Figure 7

[0186] The processor 22 implements the data transmission method provided by any of the above embodiments by invoking instructions stored in the memory 21.

[0187] For the data transmission device provided by the present application, please refer to the above method embodiments, and the present application will not be described here again, which has the same beneficial effects as the above data transmission method.

[0188] Further, the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor 22 to implement the steps of the above data transmission method.

[0189] It can be understood that if the method in the above embodiments is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and executes all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0190] For the computer readable storage medium provided by the present application, please refer to the above method embodiments, and the present application will not be described here again, which has the same beneficial effects as the above data transmission method.

[0191] The above provides a detailed description of the data processing method, device and medium provided by the present application. The embodiments in the specification are described in a progressive manner, and each embodiment mainly describes the differences from other embodiments. The same or similar parts of each embodiment can be referred to. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the present application.

[0192] ​It also needs to be explained that in the present specification, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A data processing method, characterized by, The method comprises the following steps: responding to an operation command issued by a host; determining a corresponding processing strategy based on the type of the operation command; when the processing strategy is a hardware acceleration strategy, performing acceleration processing on the operation command by a hardware acceleration unit to obtain first processing information; after reconstructing the first processing information and the operation command, sending the first processing information and the operation command to the software controller for access processing by the software controller; when the processing strategy is a software processing strategy, sending the operation command to the software controller for data processing.

2. The data processing method according to claim 1, characterized in that, The method comprises the following steps: determining the command type and the command flag of the operation command, wherein the command type comprises a data operation command and a management command; obtaining the queue depth of the storage device and a preset queue depth; when the command type is a data operation command, the command flag is a forced unit access flag, and the queue depth is greater than the preset queue depth, determining that the processing strategy is a hardware acceleration strategy.

3. The data processing method of claim 1, wherein, The method comprises the following steps: determining the command type of the operation command, wherein the command type comprises a data operation command and a management command; obtaining the current load of the software controller; obtaining the bandwidth data corresponding to the data transmission channel between the storage device and the host; when the current load exceeds a preset load, the bandwidth data is less than the remaining bandwidth data, and the command type is a data operation command, determining that the processing strategy is a hardware acceleration strategy.

4. The data processing method according to claim 2 or 3, characterized in that, The method comprises the following steps: controlling the hardware acceleration unit to receive the operation command and parse the address information of the operation command to perform memory address allocation processing to obtain corresponding first address information; obtaining the lock register address and reading the lock state; when the lock state is an idle state, occupying the lock corresponding to the lock state to complete the locking processing of the operation command; performing operation processing on the operation command after the locking processing according to the first address information to obtain the first processing information.

5. The data processing method according to claim 4, characterized in that, The method comprises the following steps: reading the starting address of the hardware acceleration unit by the host through the interface configuration space to determine the base address of the memory mapping region of the operation command; allocating a lock register in the memory mapping region according to the base address; mapping the address information of the lock register of the hardware acceleration unit to the corresponding addressing space to obtain the lock register address.

6. The data processing method according to claim 4, characterized in that, The method comprises the following steps: controlling the hardware acceleration unit to apply for a physical region page to the host according to the data information and the command information of the operation command; allocating a corresponding memory address space of the hardware acceleration unit for the physical region page; controlling the hardware acceleration unit to allocate a list of the physical region page through a direct memory access engine and embed the list into the operation command; The direct memory access engine of the hardware acceleration unit configures the source address and the target address corresponding to the operation command as the first address information according to the embedded operation command.

7. The data processing method according to claim 5, characterized in that, The first processing information is reconstructed with the operation command, including: obtaining a lock address and a lock serial number corresponding to the current operation command; obtaining the first address information allocated by the current operation command; establishing a mapping relationship among the current operation command, the lock address, the lock serial number and the first address information to determine first reconstruction information; obtaining a lock property corresponding to the previous operation command and a lock property corresponding to the current operation command; if the lock properties corresponding to the previous operation command and the current operation command are both synchronous access, performing pre-reconstruction processing on the first reconstruction information corresponding to the previous operation command and the current operation command to obtain first pre-reconstruction information; based on the first pre-reconstruction information, judging whether there is same address information in the first address information corresponding to the previous operation command and the current operation command; if there is same address information, performing reconstruction processing on the first reconstruction information corresponding to the current operation command in a different address chain table of the previous operation command and other operation commands; if there is no same address information, adding the data amount corresponding to the current operation command to the first pre-reconstruction information to complete the reconstruction processing of the current operation command and the previous operation command.

8. The data processing method according to claim 7, characterized in that, After the reconstruction processing of the first processing information and the operation command, the operation command is sent to the software controller, including: obtaining target information corresponding to the reconstructed operation command; retaining operation information corresponding to the software controller in the target information to send to the software controller; deleting operation information not corresponding to the software controller in the target information to release the corresponding memory space.

9. A data processing apparatus, characterized by, including: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the data processing method according to any one of claims 1 to 8.

10. A computer readable storage medium characterized by The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data processing method according to any one of claims 1 to 8.

Images