Data processing method based on object storage engine and related equipment

Abstract

The embodiment of the invention discloses a data processing method based on an object storage engine, which comprises the following steps: after a kernel receives an I / O (Input / Output) request of an application program, analyzing the I / O request to obtain I / O operation content, and transmitting the I / O operation content to an NVMe (Non-Volatile Memory Express) driving program; converting the I / O operation content into a hardware command by using the NVMe drive program, and transmitting the hardware command to the NVMe storage device; after an I / O operation is completed on the NVMe storage device according to the hardware command and an I / O operation result is generated, the I / O operation result is received based on the NVMe drive program; a preset kernel function is triggered on the basis of the NVMe drive program, the I / O operation result is returned to the application program on the basis of the kernel function, and the kernel function is constructed on the basis of the Berkley data packet filter. By means of the mode, the I / O operation speed can be increased, and the I / O processing performance of the system is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data storage, and in particular to a data processing method, apparatus, device and storage medium based on an object storage engine. Background Art

[0002] With the rise of high-performance storage technology, storage media has been continuously developing from SATA, SAS to NVMe. The performance of storage media is getting better and better, and the latency and power consumption are getting lower and lower. For example, NVMe storage devices can now achieve a bandwidth of up to 7GB / s and a latency as low as 3μs. The IO stack mainly manages and coordinates the process of data exchange between the operating system and the hardware storage device. It includes multiple layers, including the block device layer, the file system layer, and the system call layer. Each layer is responsible for handling specific tasks to ensure that data can be smoothly transmitted between the application and the storage device. Under such high performance of storage media, the structure of the existing IO stack makes the path of data between the operating system and the storage device longer, resulting in large data storage delays and call overhead, low rate, and system performance needs to be improved. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention provide a data processing method, apparatus, device and storage medium based on an object storage engine, which are used to solve the problem of low data storage rate in the prior art.

[0004] According to one aspect of an embodiment of the present invention, a data processing method based on an object storage engine is provided, the method comprising:

[0005] After the kernel receives the I / O request of the application, it parses the I / O request to obtain the I / O operation content, and transmits the I / O operation content to the NVMe driver;

[0006] Utilize the NVMe driver to convert the I / O operation content into a hardware command, and transmit the hardware command to the NVMe storage device;

[0007] After completing the I / O operation according to the hardware command on the NVMe storage device and generating an I / O operation result, receiving the I / O operation result based on the NVMe driver;

[0008] A predetermined kernel function is triggered based on the NVMe driver, and the I / O operation result is returned to the application based on the kernel function, wherein the kernel function is constructed based on a Berkeley packet filter.

[0009] In an optional manner, the method further comprises: pre-building a data packet processing process in the kernel, the data packet processing process comprising a kernel component;

[0010] The transmitting the I / O operation content to the NVMe driver comprises: utilizing the kernel component and transmitting the I / O operation content to the NVMe driver through a preset functional interface.

[0011] In an optional manner, before transmitting the I / O operation content to the NVMe driver, the method further includes:

[0012] Analyze whether the I / O operation content is an operation content of object storage;

[0013] If so, based on the kernel function, the metadata in the I / O operation content is obtained, the metadata is calculated multiple times based on a predetermined plurality of hash functions, and the calculation results are stored in a storage location of a key-value storage system data structure, and the key-value storage system is pre-built based on the kernel function.

[0014] In an optional manner, one or more sub-storage locations may be added to each of the storage locations, and storing the calculation results in the storage locations of the key-value storage system data structure includes:

[0015] Analyze whether there are metadata with the same calculation results;

[0016] If so, the same calculation results are taken as the same set, and the calculation results of the same set are stored in the storage location and the sub-storage location corresponding to the storage location.

[0017] In an optional manner, the method further includes:

[0018] An ordered linked list corresponding to the calculation results of the same set is constructed, and a multi-layer index of the ordered linked list is constructed, wherein each node in the multi-layer index includes an index node and a data node.

[0019] In an optional manner, the method further includes:

[0020] When the metadata is deleted or searched, or new metadata is inserted, the ordered linked list and the multiple layers of the index are updated.

[0021] In an optional manner, the performing multiple calculations on the metadata based on multiple predetermined hash functions includes:

[0022] Slicing the metadata to obtain a plurality of slice data, and performing a predetermined first hash function operation on the slice data to obtain a first hash result;

[0023] Based on the slice data, a predetermined second hash function, the number of the slice data, and the sequence number of the slice data, a second hash result is determined, and a sum of the first hash result and the second hash result is determined.

[0024] According to another aspect of an embodiment of the present invention, there is provided a data processing device based on an object storage engine, comprising:

[0025] A parsing module, configured to parse the I / O request to obtain I / O operation content after the kernel receives the I / O request of the application, and transmit the I / O operation content to the NVMe driver;

[0026] A conversion module, used to convert the I / O operation content into a hardware command using the NVMe driver, and transmit the hardware command to the NVMe storage device;

[0027] A transmission module, configured to receive the I / O operation result based on the NVMe driver after completing the I / O operation according to the hardware command on the NVMe storage device and generating the I / O operation result;

[0028] A trigger module is used to trigger a predetermined kernel function based on the NVMe driver, and return the I / O operation result to the application based on the kernel function, wherein the kernel function is constructed based on the Berkeley packet filter.

[0029] According to another aspect of an embodiment of the present invention, there is provided a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method described above.

[0030] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and when the executable instruction is executed on a computer device, the computer device executes the method described above.

[0031] In an embodiment of the present invention, after the kernel receives the I / O request of the application, the I / O operation content is directly transmitted to the NVMe driver, avoiding data from entering the traditional kernel hierarchy, including the block device layer, the file system layer and the system call layer, and is directly processed by the NVMe driver, which greatly improves the rate of I / O operations. When the I / O operation is completed, the NVMe driver triggers a predetermined kernel function to return the I / O operation result to the application, thereby constructing a new and shorter I / O operation path. At the same time, the kernel function is constructed based on the Berkeley packet filter, and can also use the existing capabilities of BPF to achieve acceleration and rapid response of I / O processing, thereby improving the I / O processing performance of the system.

[0032] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0034] Figure 1 A schematic diagram of the use environment of the data processing method based on the object storage engine provided by an embodiment of the present invention is shown;

[0035] Figure 2 A schematic diagram of a process flow of a data processing method based on an object storage engine provided by an embodiment of the present invention is shown;

[0036] Figure 3 A schematic diagram of the structure of a data processing device based on an object storage engine provided by an embodiment of the present invention is shown;

[0037] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0039] Figure 1 Shows Figure 1 The following is a schematic diagram showing the use environment of the data processing method based on the object storage engine provided by the embodiment of the present invention. Figure 1As shown, the operating system includes a kernel and an NVMe driver. The application APP initiates an I / O request. After the kernel in the operating system receives and parses the I / O request, it does not process the data through multiple layers of the kernel, but directly transmits the parsed content to the NVMe driver. The NVMe driver interacts with the NVMe storage device and implements related operations of the I / O request on the NVMe storage device. After the operation is completed, the NVMe storage device generates an interrupt and returns the operation completion signal to the application APP along the original path. This I / O operation path is a new and shorter I / O operation path. The data bypasses the traditional kernel hierarchy and increases the rate of I / O operations.

[0040] Figure 2 A flow chart showing an embodiment of a data processing method based on an object storage engine of the present invention is shown as follows: Figure 2 As shown, the method comprises the following steps:

[0041] Step 101, after the kernel receives the I / O request of the application, it parses the I / O request to obtain the I / O operation content, and transmits the I / O operation content to the NVMe driver.

[0042] In this embodiment, when an application performs an I / O operation, the application sends an I / O request to the operating system. The I / O request is an input / output operation request, and the content involved includes reading data, writing data, or network communication, etc. After receiving the I / O request, the kernel of the operating system parses it and obtains the I / O operation content, including the file or device to be accessed by this I / O request, as well as the data to be processed and the type of I / O operation. The type of I / O operation includes reading or writing.

[0043] In the related art, after the kernel receives the I / O request of the application, it will be processed by the kernel hierarchy, which includes the block device layer, the file system layer and the system call layer. The I / O request is processed by the kernel hierarchy, which makes the I / O request processing rate slow. In this embodiment, after the I / O operation content is parsed, it will not be transmitted to the kernel hierarchy for processing, but directly transmitted to the NVMe driver, thereby bypassing the processing of the kernel hierarchy. Non-Volatile Memory Express (NVMe) is a high-speed storage interface standard, and its driver can interact directly with the hardware. Therefore, in this embodiment, the I / O operation content is transmitted to the NVMe driver, and the NVMe driver is used to directly interact with the hardware device, which can accelerate the I / O processing rate.

[0044] Optionally, this embodiment pre-builds an independent data packet processing process in the kernel, and the data packet processing process includes a kernel component, which is a customized kernel component designed according to processing requirements, wherein transmitting the I / O operation content to the NVMe driver includes: using the kernel component and transmitting the I / O operation content to the NVMe driver through a preset functional interface. The kernel component of this embodiment closely cooperates with the NVMe driver through a preset functional interface to directly process the received data, so that the I / O operation content can be directly unloaded to the underlying NVMe storage device through the NVMe driver, without the need to allocate software queues in advance, reducing unnecessary intermediate links and delays, and improving the I / O processing rate.

[0045] Step 102, using the NVMe driver to convert the I / O operation content into a hardware command, and transmit the hardware command to the NVMe storage device.

[0046] In this embodiment, after the I / O operation content reaches the NVMe driver, the NVMe driver converts it into a hardware command based on the file or device to be accessed this time, as well as the type of data to be processed and the I / O operation, and the NVMe driver directly transmits the hardware command to the NVMe storage device.

[0047] Step 103, after completing the I / O operation according to the hardware command on the NVMe storage device and generating the I / O operation result, receiving the I / O operation result based on the NVMe driver.

[0048] In this embodiment, after the NVMe storage device completes the I / O operation according to the hardware command, the NVMe storage device will send an interrupt signal to the CPU. At this time, the CPU will pause the currently executing program and execute the interrupt handler to respond to the interrupt signal. The interrupt handler is a program used by the operating system to respond to hardware interrupts.

[0049] Step 104: trigger a predetermined kernel function based on the NVMe driver, and return the I / O operation result to the application based on the kernel function, wherein the kernel function is constructed based on the Berkeley packet filter.

[0050] In this embodiment, the NVMe driver triggers the kernel function after receiving the I / O operation result. The kernel function can be executed immediately after the I / O operation is completed to process the result of the I / O operation or perform subsequent operations, thereby improving the response speed of the system. Specifically, the kernel function returns the I / O operation result to the application. If the I / O operation result is a successful operation, the application can continue to execute; if the I / O operation result is a failed operation, the application can perform other corresponding operations based on the failure information.

[0051] Among them, the Berkeley Packet Filter (BPF) provides a capability to execute a section of code when a specific event of the kernel or application occurs. It adopts the virtual machine instruction specification so that it can provide a capability to extend the kernel without modifying the kernel source code and recompiling, without modifying the core kernel, and maintaining system stability and code security. In the operating system Linux, the kernel's support for BPF allows it to use the existing kernel and file system without the need for special hardware or major changes to the application. In addition, BPF works in the kernel, avoiding the copying of data from the kernel state to the user state, and has a fast speed. Therefore, this embodiment constructs a kernel function based on the Berkeley Packet Filter and expands it into a general execution engine to accelerate the application I / O data processing and improve the I / O data processing efficiency.

[0052] Among them, building a kernel function based on the Berkeley packet filter and expanding it into an execution engine includes: creating a custom kernel module (kernel function) to ensure that it can be safely exposed to the BPF program for use, creating a BPF program, which can use the custom kernel module to perform related operations in the above-mentioned I / O path, and finally compiling, loading and running the kernel module and BPF program to implement the operations of the above-mentioned I / O path.

[0053] In this embodiment, after the kernel receives the I / O request of the application, the I / O operation content is directly transmitted to the NVMe driver, so as to avoid data entering the traditional kernel hierarchy, including the block device layer, the file system layer and the system call layer. The data is directly processed by the NVMe driver, which greatly improves the rate of I / O operations. When the I / O operation is completed, the NVMe driver triggers a predetermined kernel function to return the I / O operation result to the application, thereby constructing a new and shorter I / O operation path. At the same time, the kernel function is constructed based on the Berkeley packet filter, and can also use the existing capabilities of BPF to achieve acceleration and rapid response of I / O processing, thereby improving the I / O processing performance of the system.

[0054] Optionally, before transmitting the I / O operation content to the NVMe driver, it also includes: analyzing whether the I / O operation content is the operation content of object storage; if so, based on the kernel function, obtaining metadata in the I / O operation content, performing multiple calculations on the metadata based on a predetermined plurality of hash functions, and storing the calculation results in a storage location of a key-value storage system data structure, wherein the key-value storage system is pre-built based on the kernel function.

[0055] In this embodiment, object storage is a structured method of data management and storage based on objects as basic units. Each object has a unique identifier Key, and contains the data itself and metadata related to it. Metadata is the descriptive information of the object, which provides additional information about the object, such as the creation time, modification time, size, type, etc. of the object, including system-defined metadata and user-defined metadata. System-defined metadata includes the name, version, size, and hash value of the object, etc. User-defined metadata is usually any descriptive information saved in the form of key-value pairs, such as the time and place of taking a photo, the author and singer of a song, etc. In this embodiment, if the I / O operation content is the operation content of object storage, the kernel function is constructed based on the Berkeley packet filter and expanded into an object storage engine.

[0056] In the prior art, data is generally stored using a single hash operation, which is prone to hash collisions. The key-value storage system of this embodiment achieves the function of reducing hash collisions and improving storage utilization. Specifically, when a piece of data is inserted into the key-value storage system for storage, metadata is calculated multiple times based on multiple predetermined hash functions, and the calculation results are stored in the storage location of the key-value storage system data structure. This embodiment performs multiple hash calculations on metadata, making the storage location of metadata more dispersed, thereby reducing the occurrence of hash collisions and improving the utilization of metadata storage space.

[0057] The number of calculations is not limited here, and can be two or more times, each time using a different hash function. Optionally, this embodiment performs a secondary hash calculation, including: slicing the metadata to obtain multiple slice data, performing a predetermined first hash function operation on the slice data to obtain a first hash result; determining a second hash result based on the slice data, a predetermined second hash function, the number of slice data, and the sequence number of the slice data, and determining the sum of the first hash result and the second hash result.

[0058] In this embodiment, the metadata P is sliced ​​to obtain multiple slice data p1, p2, p3...pn, where n is the number of slice data, a predetermined first hash function is performed on the slice data to obtain a first hash result Hash1(pi), i is the serial number of the slice data, a predetermined second hash function is performed on the slice data to obtain Hash2(pi), and a remainder result Hash2(pi)%n is determined based on Hash2(pi) and the number of slice data n, and the product of the remainder result and the serial number of the slice data i*Hash2(pi)%n is obtained to obtain a second hash result, and the second hash result is added to the first hash result.

[0059] In an optional embodiment, one or more sub-storage locations may be added to each of the storage locations, and the storage of the calculation results in the storage location of the key-value storage system data structure includes: analyzing whether there are identical calculation results of the metadata; if so, treating the identical calculation results as the same set, and storing the calculation results of the same set in the storage location and the sub-storage location corresponding to the storage location.

[0060] In this embodiment, the storage location of the metadata is the list address, that is, multiple data with the same calculation results under the same hash position can be stored continuously in the memory, taking advantage of the CPU cache and further improving the utilization rate of the storage space and the access speed.

[0061] In an optional embodiment, the method further includes: constructing a corresponding ordered linked list of the calculation results of the same set, and constructing a multi-layer index of the ordered linked list, wherein each node in the multi-layer index includes an index node and a data node.

[0062] In this embodiment, a multi-layer index is constructed by combining the concepts of linked lists and indexes, aiming to achieve efficient data retrieval and storage. Among them, the bottom index node is constructed based on an ordered linked list, and each index node corresponds to a storage location or sub-storage location in the ordered linked list. The index nodes at the beginning of each layer of index are the same, and the index nodes of the upper index are less than the index nodes of the lower index. The index node is a special node used to record the location information of its node. It does not store actual data, but points to the data node. The index node usually contains a pointer (pointing to the next index node or data node) and a keyword (an identifier for quickly locating the data node). The data node stores the actual data and contains a pointer to the next data node. All nodes in the ordered linked list can be arranged in a certain order, usually in ascending or descending order. This orderliness not only enhances the compactness of data storage, but also significantly improves the cache hit rate. When querying data in large and small intervals, since the data has been arranged in a certain order, it is possible to retrieve the required information more efficiently without checking each element one by one like unordered data. In addition, each node contains not only data but also a pointer to the next node. This structure allows for bidirectional indexing, i.e., each index node can traverse the ordered linked list forward and backward to improve search efficiency. In addition, index nodes can be dynamically inserted or deleted as needed. To further improve performance, the ordered linked list can be divided into multiple blocks, each of which is managed by an index node, reducing the pressure on a single index node.

[0063] In an optional embodiment, the method further includes: when deleting or searching the metadata, or inserting new metadata, updating the ordered linked list and the multiple layers of the index.

[0064] When deleting or searching for the metadata, or inserting new metadata, the multi-layer index is updated: the insertion operation needs to find a suitable position, insert the new node into the ordered linked list, and update the information of the related index nodes in the multi-layer index. The deletion operation needs to retrieve the node to be deleted from the ordered linked list, remove it from the ordered linked list, and update the information of the related index nodes in the multi-layer index. The search operation can quickly locate the target node through the index node, and then traverse along the linked list until the required data is found.

[0065] Figure 3 FIG. 1 shows a schematic diagram of the structure of a data processing device based on an object storage engine. Figure 3 As shown, the data processing device 300 of the object storage engine includes:

[0066] The parsing module 301 is used for parsing the I / O request to obtain the I / O operation content after the kernel receives the I / O request of the application, and transmitting the I / O operation content to the NVMe driver;

[0067] A conversion module 302 is used to convert the I / O operation content into a hardware command using the NVMe driver, and transmit the hardware command to the NVMe storage device;

[0068] A transmission module 303 is used for receiving the I / O operation result based on the NVMe driver after completing the I / O operation according to the hardware command on the NVMe storage device and generating the I / O operation result;

[0069] The trigger module 304 is used to trigger a predetermined kernel function based on the NVMe driver, and return the I / O operation result to the application based on the kernel function, wherein the kernel function is constructed based on the Berkeley packet filter.

[0070] In an embodiment of the present invention, after the kernel receives the I / O request of the application, the I / O operation content is directly transmitted to the NVMe driver, avoiding data from entering the traditional kernel hierarchy, including the block device layer, the file system layer and the system call layer, and is directly processed by the NVMe driver, which greatly improves the rate of I / O operations. When the I / O operation is completed, the NVMe driver triggers a predetermined kernel function to return the I / O operation result to the application, thereby constructing a new and shorter I / O operation path. At the same time, the kernel function is constructed based on the Berkeley packet filter, and can also use the existing capabilities of BPF to achieve acceleration and rapid response of I / O processing, thereby improving the I / O processing performance of the system.

[0071] Figure 4The schematic diagram of the structure of the computer device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0072] like Figure 4 As shown, the computer device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0073] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other computer devices such as client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps used in the above embodiment.

[0074] Specifically, the program 410 may include program code including computer executable instructions.

[0075] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0076] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0077] The program 410 may be specifically called by the processor 402 to enable the computer device to perform the following operations:

[0078] After the kernel receives the I / O request of the application, it parses the I / O request to obtain the I / O operation content, and transmits the I / O operation content to the NVMe driver;

[0079] Utilize the NVMe driver to convert the I / O operation content into a hardware command, and transmit the hardware command to the NVMe storage device;

[0080] After completing the I / O operation according to the hardware command on the NVMe storage device and generating an I / O operation result, receiving the I / O operation result based on the NVMe driver;

[0081] A predetermined kernel function is triggered based on the NVMe driver, and the I / O operation result is returned to the application based on the kernel function, wherein the kernel function is constructed based on a Berkeley packet filter.

[0082] In an optional manner, it further includes: pre-building a data packet processing process in the kernel, the data packet processing process including a kernel component;

[0083] The transmitting the I / O operation content to the NVMe driver comprises: utilizing the kernel component and transmitting the I / O operation content to the NVMe driver through a preset functional interface.

[0084] In an optional manner, before transmitting the I / O operation content to the NVMe driver, the method further includes:

[0085] Analyze whether the I / O operation content is an operation content of object storage;

[0086] If so, based on the kernel function, the metadata in the I / O operation content is obtained, the metadata is calculated multiple times based on a predetermined plurality of hash functions, and the calculation results are stored in a storage location of a key-value storage system data structure, and the key-value storage system is pre-built based on the kernel function.

[0087] In an optional manner, one or more sub-storage locations may be added to each of the storage locations, and storing the calculation results in the storage locations of the key-value storage system data structure includes:

[0088] Analyze whether there are metadata with the same calculation results;

[0089] If so, the same calculation results are taken as the same set, and the calculation results of the same set are stored in the storage location and the sub-storage location corresponding to the storage location.

[0090] In an optional manner, the method further includes:

[0091] An ordered linked list corresponding to the calculation results of the same set is constructed, and a multi-layer index of the ordered linked list is constructed, wherein each node in the multi-layer index includes an index node and a data node.

[0092] In an optional manner, the method further includes:

[0093] When the metadata is deleted or searched, or new metadata is inserted, the ordered linked list and the multiple layers of the index are updated.

[0094] In an optional manner, the performing multiple calculations on the metadata based on multiple predetermined hash functions includes:

[0095] Slicing the metadata to obtain a plurality of slice data, and performing a predetermined first hash function operation on the slice data to obtain a first hash result;

[0096] Based on the slice data, a predetermined second hash function, the number of the slice data, and the sequence number of the slice data, a second hash result is determined, and a sum of the first hash result and the second hash result is determined.

[0097] In an embodiment of the present invention, after the kernel receives the I / O request of the application, the I / O operation content is directly transmitted to the NVMe driver, avoiding data from entering the traditional kernel hierarchy, including the block device layer, the file system layer and the system call layer, and is directly processed by the NVMe driver, which greatly improves the rate of I / O operations. When the I / O operation is completed, the NVMe driver triggers a predetermined kernel function to return the I / O operation result to the application, thereby constructing a new and shorter I / O operation path. At the same time, the kernel function is constructed based on the Berkeley packet filter, and can also use the existing capabilities of BPF to achieve acceleration and rapid response of I / O processing, thereby improving the I / O processing performance of the system.

[0098] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device, the computer device executes any of the above method embodiments, including:

[0099] After the kernel receives the I / O request of the application, it parses the I / O request to obtain the I / O operation content, and transmits the I / O operation content to the NVMe driver;

[0100] Utilize the NVMe driver to convert the I / O operation content into a hardware command, and transmit the hardware command to the NVMe storage device;

[0101] After completing the I / O operation according to the hardware command on the NVMe storage device and generating an I / O operation result, receiving the I / O operation result based on the NVMe driver;

[0102] A predetermined kernel function is triggered based on the NVMe driver, and the I / O operation result is returned to the application based on the kernel function, wherein the kernel function is constructed based on a Berkeley packet filter.

[0103] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computer device to execute any of the above method embodiments.

[0104] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer is enabled to execute any of the above method embodiments.

[0105] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description made to specific languages ​​above is for disclosing the best mode of the present invention.

[0106] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0107] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly recited in each claim.

[0108] Those skilled in the art will appreciate that the modules in the computer device in the embodiment can be adaptively changed and set in one or more computer devices different from the embodiment. The modules or units or components in the embodiment can be combined into one module or unit or component, and they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or computer device disclosed in this manner can be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0109] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data processing method based on an object storage engine, characterized in that: The method comprises: After the kernel receives the I / O request of the application, it parses the I / O request to obtain the I / O operation content, and transmits the I / O operation content to the NVMe driver; Utilize the NVMe driver to convert the I / O operation content into a hardware command, and transmit the hardware command to the NVMe storage device; After completing the I / O operation according to the hardware command on the NVMe storage device and generating an I / O operation result, receiving the I / O operation result based on the NVMe driver; A predetermined kernel function is triggered based on the NVMe driver, and the I / O operation result is returned to the application based on the kernel function, wherein the kernel function is constructed based on a Berkeley packet filter.

2. The method according to claim 1, characterized in that The method further comprises: pre-building a data packet processing process in the kernel, the data packet processing process comprising a kernel component; The transmitting the I / O operation content to the NVMe driver comprises: utilizing the kernel component and transmitting the I / O operation content to the NVMe driver through a preset functional interface.

3. The method according to claim 1, characterized in that: Before transmitting the I / O operation content to the NVMe driver, the method further includes: Analyze whether the I / O operation content is an operation content of object storage; If so, based on the kernel function, the metadata in the I / O operation content is obtained, the metadata is calculated multiple times based on a predetermined plurality of hash functions, and the calculation results are stored in a storage location of a key-value storage system data structure, and the key-value storage system is pre-built based on the kernel function.

4. The method according to claim 3, characterized in that One or more sub-storage locations may be added to each of the storage locations, and the calculation results are stored in the storage locations of the key-value storage system data structure, including: Analyze whether there are metadata with the same calculation results; If so, the same calculation results are taken as the same set, and the calculation results of the same set are stored in the storage location and the sub-storage location corresponding to the storage location.

5. The method according to claim 4, characterized in that The method further comprises: An ordered linked list corresponding to the calculation results of the same set is constructed, and a multi-layer index of the ordered linked list is constructed, wherein each node in the multi-layer index includes an index node and a data node.

6. The method according to claim 5, characterized in that The method further comprises: When the metadata is deleted or searched, or new metadata is inserted, the ordered linked list and the multiple layers of the index are updated.

7. The method according to claim 3, characterized in that The performing multiple calculations on the metadata based on multiple predetermined hash functions includes: Slicing the metadata to obtain a plurality of slice data, and performing a predetermined first hash function operation on the slice data to obtain a first hash result; Based on the slice data, a predetermined second hash function, the number of the slice data, and the sequence number of the slice data, a second hash result is determined, and a sum of the first hash result and the second hash result is determined.

8. A data processing device based on an object storage engine, characterized in that: include: A parsing module, configured to parse the I / O request to obtain I / O operation content after the kernel receives the I / O request of the application, and transmit the I / O operation content to the NVMe driver; A conversion module, used to convert the I / O operation content into a hardware command using the NVMe driver, and transmit the hardware command to the NVMe storage device; A transmission module, configured to receive the I / O operation result based on the NVMe driver after completing the I / O operation according to the hardware command on the NVMe storage device and generating the I / O operation result; A trigger module is used to trigger a predetermined kernel function based on the NVMe driver, and return the I / O operation result to the application based on the kernel function, wherein the kernel function is constructed based on the Berkeley packet filter.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device, the computer device executes the method according to any one of claims 1 to 7.

Images