Data processing method and device

By introducing cache logs and asynchronous write mechanisms into NFS servers, the latency and consistency problems of traditional NFS servers in high concurrent write scenarios are solved, which improves performance and reliability, and reduces hardware costs.

CN118921358AActive Publication Date: 2024-11-08CHINA CONSTRUCTION BANK +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410921553.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-08
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

Traditional NFS servers face write latency and data consistency problems in high concurrent write scenarios, especially in terms of crash recovery and hardware resource utilization, resulting in performance bottlenecks and high costs.

Method used

Introduce cache logs as the intermediate cache layer, asynchronously write logs to the log disk and return a response, and then asynchronously write data to the back-end storage system. The combination of NVMe log disk and ordinary mechanical disk is used to achieve rapid recording and persistent writing. Enter requests to ensure data consistency and high performance.

Benefits of technology

It significantly improves the write performance of NFS gateways, shortens I/O delay, reduces hardware costs, ensures data consistency and system reliability, and simplifies the failure recovery process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921358A_ABST
    Figure CN118921358A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and device, and relates to the technical field of data processing in the field of big data. A specific embodiment of the method comprises the following steps: receiving a data processing request transmitted by an NFS client through a communication layer, and sending the data processing request to a protocol layer; a request type and a request parameter are obtained through analysis of the protocol layer, the request type is a write request, the request parameter is written into a log disk of the log layer in a log form, and a log record is obtained; after writing is successful, response success information of the NFS client is returned through the communication layer; asynchronously reading a log record from a log disk and sending the log record to the interface layer through a log processing module of the log layer; and determining a storage position in the back-end storage system through the interface layer according to the request parameter, and calling a driving mode corresponding to the storage position in the driving layer so as to write the to-be-written data in the request parameter into the storage position. The implementation mode has the characteristics of high read-write performance and low delay, and the write-in performance of the NFS gateway can be remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology in the field of big data, and in particular to a data processing method and device. Background Art

[0002] With the rapid development of big data and cloud computing, distributed file systems are playing an increasingly important role in enterprise-level storage solutions. As a mature and widely used distributed file system protocol, NFS (full name: Network File System) is widely used in various application scenarios. However, traditional NFS servers often face performance bottlenecks and data consistency issues in high-concurrency and large-data volume scenarios.

[0003] In order to improve the performance and reliability of the NFS server, the NFS gateway, as a user space NFS server, provides a flexible architecture and rich functional support. However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: in a high-concurrency write scenario, the NFS gateway may still face problems of write delay and data consistency. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data processing method and device, which can at least solve the problems of write delay and data consistency faced by the prior art in a high-concurrency write scenario.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, which is applied to a distributed file system protocol NFS server, and the NFS server includes a communication layer, a protocol layer, a log layer, an interface layer and a driver layer, including:

[0006] Receive a data processing request transmitted by the NFS client through the communication layer, and send the data processing request to the protocol layer;

[0007] Parse the data processing request through the protocol layer to obtain the request type and request parameters. If the request type is a write request, write the request parameters into the log disk of the log layer in the form of logs to obtain log records.

[0008] After the writing is successful, the NFS client is returned a response success message through the communication layer; and the log processing module of the log layer asynchronously reads the log record from the log disk and sends it to the interface layer;

[0009] The interface layer determines the storage location in the backend storage system according to the request parameters in the log record, and calls the driver mode corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0010] Optionally, the data processing request further includes NFS protocol version information, and a corresponding processing service is determined according to the NFS protocol version information.

[0011] Optionally, the request parameters include a file name, an offset, a length, the data to be written, and file attributes; determining a storage location in a backend storage system according to the request parameters in the log record, and calling a driver mode corresponding to the storage location in a driver layer to write the data to be written in the request parameters into the storage location, includes:

[0012] Determine the corresponding storage node from the backend storage system according to the file name;

[0013] According to the offset and length, determine the corresponding storage location from the storage node;

[0014] A driving mode corresponding to the storage node in the driving layer is called to write the data to be written and the file attribute into the storage location of the storage node.

[0015] Optionally, the method further comprises:

[0016] Marking the state of the log record written to the log disk as a committed state;

[0017] After the calling interface layer and the driver layer write the data to be written in the log record into the backend storage system, the state of the log record is updated from the submitted state to the applied state;

[0018] Triggers the clearing operation of log records in the log disk whose status is application status.

[0019] Optionally, the method further comprises:

[0020] In response to the NFS server failure restart operation, check whether there is a log record in the log layer with a status of commit;

[0021] In response to the check result being existence, the interface layer and the driver layer are called to write the to-be-written data in the log record into the backend storage system.

[0022] Optionally, the NFS server further includes a data cache layer, and the method further includes:

[0023] When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cache data is queried from the data cache layer according to the file name, offset, and length;

[0024] In response to cache data misses, the corresponding storage node and storage location are determined from the backend storage system according to the file name, offset and length, and then the data is read from the storage location of the storage node and returned to the client.

[0025] To achieve the above object, according to another aspect of an embodiment of the present invention, a data processing device is provided, which is applied to a distributed file system protocol NFS server, including:

[0026] The communication module is used to receive a data processing request transmitted by the NFS client and send the data processing request to the protocol layer;

[0027] A protocol module, used to parse the data processing request, obtain the request type and request parameters, and when the request type is a write request, write the request parameters into a log disk of the log layer in the form of a log to obtain a log record;

[0028] The log processing module is used to return the NFS client response success information through the communication layer after the writing is successful; and asynchronously read the log record from the log disk and send it to the interface layer;

[0029] The interface driver module is used to determine the storage location in the backend storage system through the interface layer according to the request parameters in the log record, and call the driving method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0030] Optionally, the data processing request further includes NFS protocol version information, and a corresponding processing service is determined according to the NFS protocol version information.

[0031] Optionally, the request parameters include a file name, an offset, a length, and the interface driver module is used to:

[0032] Determine the corresponding storage node from the backend storage system according to the file name;

[0033] According to the offset and length, determine the corresponding storage location from the storage node;

[0034] A driving mode corresponding to the storage node in the driving layer is called to write the data to be written and the file attribute into the storage location of the storage node.

[0035] Optionally, the device further comprises a state module, configured to:

[0036] Marking the state of the log record written to the log disk as a committed state;

[0037] After the calling interface layer and the driver layer write the data to be written in the log record into the backend storage system, the state of the log record is updated from the submitted state to the applied state;

[0038] Triggers the clearing operation of log records in the log disk whose status is application status.

[0039] Optionally, the device further comprises a recovery module, configured to:

[0040] In response to the NFS server failure restart operation, check whether there is a log record in the log layer with a status of commit;

[0041] In response to the check result being existence, the interface layer and the driver layer are called to write the to-be-written data in the log record into the backend storage system.

[0042] Optionally, the NFS server further includes a data cache layer, and the device further includes a reading module, which is used to:

[0043] When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cache data is queried from the data cache layer according to the file name, offset, and length;

[0044] In response to cache data misses, the corresponding storage node and storage location are determined from the backend storage system according to the file name, offset and length, and then the data is read from the storage location of the storage node and returned to the client.

[0045] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing electronic device is provided.

[0046] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data processing methods.

[0047] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above data processing methods is implemented.

[0048] To achieve the above object, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product of an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present invention is implemented.

[0049] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: introducing a cache log as an intermediate cache layer for quickly recording write request log records. After writing to the log disk successfully, a successful response message is immediately returned to the client, and then the process of writing data to the back-end storage system is performed asynchronously. This asynchronous operation can shorten the I / O path, reduce I / O latency, improve write performance, and solve the existing write delay problem. And because the data is written to the log disk, there is no pollution of the back-end data, which solves the existing data inconsistency problem. Because only the log disk uses NVMe, the back-end data disk can use an ordinary mechanical disk. The combination of the two gives full play to the hardware characteristics and reduces hardware costs. The cache log of the present application has the characteristics of high read and write performance and low latency, which can significantly improve the write performance of the NFS gateway.

[0050] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0052] Figure 1 is a main flow chart of a data processing method according to an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of the NFS gateway structure of an embodiment of the present invention;

[0054] Figure 3 is a flowchart of an optional data processing method according to an embodiment of the present invention;

[0055] Figure 4 is a flowchart of a specific data processing method according to an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0057] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0058] Figure 7 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0060] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0061] In the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The acquisition, transmission, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations, and are used for legal and reasonable purposes, and are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities.

[0062] For user information, necessary measures should be taken to prevent illegal access to such personal information data, ensure that persons who have access to personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting data. When applicable, including in certain relevant applications, protect user privacy by de-identifying data, such as by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the specific address level), controlling how data is stored, and / or other methods of de-identification.

[0063] Here are the shortcomings of the prior art described in detail:

[0064] 1. Write performance bottleneck: 1) High write latency: When processing write operations, traditional NFS usually needs to wait for data to be written to the backend storage system, which will result in high write latency. Especially in a high-concurrency environment, write latency will increase significantly, affecting overall system performance. 2) Blockage caused by synchronous write: Many traditional NFS implementations use synchronous write mode, that is, each write request needs to wait for the write operation to complete before returning a completion message to the client. This synchronous write mode is prone to request blocking, affecting the system's response time and concurrent processing capabilities.

[0065] 2. Data consistency issues. 1) Difficulties in crash recovery: After a system crash or unexpected restart, traditional NFS systems have difficulty in ensuring data consistency and integrity. Due to the lack of an effective logging mechanism, some data may be lost or data inconsistency may occur. 2) Weak consistency model: The weak consistency model of traditional NFS (such as the asynchronous write mode of NFSv3) may not meet strict data consistency requirements in some application scenarios, which poses a challenge to applications with high data consistency requirements (such as database applications).

[0066] 3. Insufficient utilization of hardware resources. High cost and low efficiency: Traditional NFS systems have problems with hardware resource utilization, especially under high-performance storage requirements, where hardware costs are high and efficiency is low. For application scenarios that require high IOPS (input and output operations per second) and low latency, traditional hard disks may not be able to meet the requirements.

[0067] See also Figure 1 , which shows a main flow chart of a data processing method provided by an embodiment of the present invention, comprising the following steps:

[0068] S101: receiving a data processing request transmitted by an NFS client through a communication layer, and sending the data processing request to a protocol layer;

[0069] S102: parsing the data processing request through the protocol layer to obtain a request type and request parameters. When the request type is a write request, writing the request parameters into a log disk of the log layer in the form of a log to obtain a log record.

[0070] S103: after the writing is successful, the NFS client is returned a response success message through the communication layer; and asynchronously, through the log processing module of the log layer, the log record is read from the log disk and sent to the interface layer;

[0071] S104: determining a storage location in the backend storage system according to the request parameters in the log record through the interface layer, and calling a driver mode corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0072] In the above implementation, NFS gateway: a device used to connect different network storages, whose main function is to convert NFS protocol into other protocols, such as CephFS or HDFS protocol, so that different storages can provide services in NFS service mode. NFS gateway can realize transparent cross-platform access to file systems, improving the flexibility and scalability of data sharing.

[0073] For steps S101 to S104, the NFS gateway (i.e., NFS server) of this solution is responsible for receiving and parsing the I / O request of the NFS client, recording the I / O request in the NFS protocol into the log layer, and returning the NFS client response success information after the log record is successfully written. The log processing module of the log layer asynchronously reads the log record and sends it to the Backend interface layer for processing. The NFS gateway consists of the RPC (Remote Procedure Call) communication layer, the NFS protocol layer, the data cache and log layer, the Backend interface layer, and the driver layer. See Figure 2 shown.

[0074] 1. RPC communication layer. The NFS client initiates I / O requests to the NFS gateway, including file read and write operations. The RPC communication layer is responsible for receiving I / O requests from the NFS client. Since NFS communication uses the RPC protocol, communication with the client requires the support of the RPC communication layer.

[0075] 2. NFS protocol layer. The RPC communication layer forwards the I / O request to the NFS protocol layer for processing. The NFS protocol layer parses the I / O request and obtains the request type, NFS protocol version information, and request parameters. The request type is a read request or a write request. A read request means that the client wants to read file data from the server, while a write request is to write data to a file on the server. During the parsing process, the NFS protocol layer also identifies the NFS protocol version used by the client request. Different versions of the NFS protocol differ in functionality, security, and performance. For example, NFSv4 introduces a stronger security model (such as Kerberos support) compared to NFSv3, and adds Compound operations to reduce the number of network round trips. Based on the NFS protocol version information, the NFS server will decide whether to use NFSv3 or NFSv4 to process the request. This choice is usually determined by the server configuration and the capabilities of the client request. If both parties support and are configured to use NFSv4, NFSv4 will be selected to provide more advanced functions. Conversely, if the client or server only supports NFSv3 or is configured to use NFSv3 first, NFSv3 will be used to process the request.

[0076] In the case where the request is a read request, the data that meets the current request is queried from the data cache. If the cache hits, the cached data that hits is returned. However, if the cache does not hit, the data that meets the current request is queried from the backend storage system through the Backend interface layer and returned. In the case where the request is a write request, the NFS protocol layer writes the request information in the form of a log to the NVMe log disk of the log layer. This step ensures the rapid persistence of write requests and reduces the risk of data loss due to network or system failures. Among them, NVMe log disk: refers to a high-speed solid-state storage disk that uses the NVMe interface standard, which is often used to store system logs, database transaction logs, and other application scenarios that are sensitive to write speed and latency.

[0077] 3. Data cache and log layer. In the NFS protocol, an I / O request (whether a read request or a write request) usually contains the following key information, which is essential for locating and processing file data:

[0078] 1) File Name: The unique name of the remote file that the client wants to access. The file name helps the server locate the specific file resource.

[0079] 2) Offset: Specifies the starting position of a read or write operation in a file. For a read request, it tells the server which byte of the file to start reading data from; for a write request, it indicates where the data should be written to the file. The offset ensures the precise location of the data operation.

[0080] 3) Length: This specifies the size of the data block to be read or written. In a read request, it defines how much data should be read starting from the specified offset; in a write request, it specifies the amount of data to be written. This helps control the amount of data transferred per I / O operation.

[0081] 4) Data: In a write request, the actual data to be written to the file will be included in the request. This part of the data is sent to the server along with the request so that the server can write it to the specified location of the file.

[0082] 5) File Attributes: Although not included in every I / O request, in some cases (especially metadata update operations), the request may carry attribute information about the file, such as modification time, permissions, ownership, etc. This information is used to update the status of the file or verify the legitimacy of the operation.

[0083] The log layer is used to store the I / O request log records received by the NFS gateway, providing high-performance writing and reading capabilities. Combining this information, the NFS gateway can accurately execute the client's read and write requests to ensure the correct processing and storage of data. After the log is written successfully, the NFS gateway returns a successful response message to the NFS client, which can be returned through the RPC communication layer. In this way, the existing problem of "many traditional NFS implementations use synchronous writing, that is, each write request needs to wait for the write operation to complete before returning a completion message to the client" is solved.

[0084] Subsequently, the log processing module is notified, and the system asynchronously reads log records from the NVMe log disk and sends the log records to the Backend interface layer, so that the actual data can be written to the backend storage system through the Backend interface layer. This asynchronous operation effectively reduces the real-time write pressure of the NFS service, improves the concurrent processing capability of the overall system, and solves the existing problem that "traditional NFS usually needs to wait for data to be written to the backend storage system when processing write operations, which will cause high write latency." After the data is successfully written to the backend storage system, the corresponding log records can be deleted from the NVMe log disk to free up storage space.

[0085] 4. The Backend interface layer and driver layer are responsible for converting the NFS I / O protocol into requests that can be received by the specific backend storage system storage, and sending them to the relevant distributed file system cluster through the Backend driver. The I / O request initiated by the NFS client is in the standard format of the NFS protocol. The Backend interface layer first receives log records through the log interface layer, and converts these NFS protocol format requests into a format that the backend storage system can understand. This is because the backend storage system (such as HDFS, Ceph, GlusterFS, etc.) usually has its own set of communication protocols and APIs (Application Programming Interface), which are not directly compatible with the NFS protocol. The conversion is not only at the protocol level, but also includes the adaptation and encapsulation of request parameters. For example, the file offset, length and other information in the NFS request are mapped to the corresponding parameters of the backend storage system to ensure that the content and intent of the request can be correctly conveyed.

[0086] In the NFS server architecture, the "Backend interface layer" plays the role of a bridge. The backend storage system is often composed of multiple nodes, each of which is responsible for storing a part of the data. The Backend interface layer also needs to decide which storage location of which specific storage node or nodes to route the request to based on the file name, offset, and length. In order to interact with different storage nodes, the Backend interface layer relies on specialized Backend drivers, such as CephfsBackend driver, VFSBackend driver, GlusterBackend driver, and NVMe driver. These drivers are software modules for specific storage nodes, providing interfaces and logic for interacting with the system. The Backend interface layer sends the converted and encapsulated data to be written and file attributes to a specific storage node by calling the corresponding driver method. The work of the Backend interface layer ensures the transparency of the NFS protocol, allowing NFS clients to seamlessly interact with the underlying diverse distributed file storage systems without directly understanding the complex details of these systems.

[0087] The method provided in the above embodiment introduces a cache log, namely an NVMe log disk, as an intermediate cache layer for quickly recording write request log records. After writing to the log disk successfully, a successful response message is immediately returned to the client, and then the process of writing data to the back-end storage system is performed asynchronously. This asynchronous operation can shorten the I / O path, reduce I / O latency, improve write performance, and solve the existing write latency problem. And because the data is written to the log disk, there is no pollution of the back-end data, which solves the problem of data inconsistency. Because only the log disk uses NVMe, the back-end data disk can use an ordinary mechanical disk. The combination of the two gives full play to the hardware characteristics and reduces hardware costs. The cache log of the present application has the characteristics of high read and write performance and low latency, which can significantly improve the write performance of the NFS gateway.

[0088] See also Figure 3 , shows a schematic flow chart of an optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0089] S301: receiving a data processing request transmitted by an NFS client through a communication layer, and sending the data processing request to a protocol layer;

[0090] S302: parsing the data processing request through the protocol layer to obtain the request type and request parameters. When the request type is a write request, writing the request parameters into the log disk of the log layer in the form of logs to obtain log records, and marking the status of the log records as a submitted status;

[0091] S303: after the writing is successful, the NFS client is returned a response success message through the communication layer; and asynchronously, through the log processing module of the log layer, the log record is read from the log disk and sent to the interface layer;

[0092] S304: determining a storage location in the backend storage system according to the request parameters in the log record through the interface layer, and calling a driver mode corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location;

[0093] S305: updating the state of the log record in the log layer from the submission state to the application state, and triggering a clearing operation of the log record in the log disk whose state is the application state;

[0094] S306: In response to the fault restart operation of the NFS server, check whether there is a log record in the log layer with a status of commit;

[0095] S307: In response to the check result being existence, the interface layer and the driver layer are called to write the to-be-written data in the log record into the backend storage system.

[0096] In the above implementation, for steps S301 to S307, during the entire I / O request processing process, the writing process from the NFS client to the NVMe log disk is a synchronous call, and the writing process from the NVMe log disk to the backend storage system data disk is an asynchronous call.

[0097] In case of successful writing to the NVMe log disk, the log records successfully written by the client to the NVMe log disk will be marked as committed. The log processing module will periodically read the log records with committed status from the NVMe log disk, write the data in the log records to the backend data disk, and change the log record status in the log disk to apply status. The records in the log disk will not be deleted at this time. When the log disk capacity threshold is reached, such as 100 log records, the log records marked as apply status on the log disk will be deleted.

[0098] In the case of failure to write to the NVMe log disk, when the client fails to write to the NVMe log disk, the failure record will not be recorded in the NVMe log disk. The client will receive the result of the write failure, and the client should initiate the write operation again. Therefore, even if the system crashes, it only applies to the NVMe log disk to the backend storage system data disk. Since the client's write operations are recorded in the NVMe log disk but not in the backend data disk, after the system restarts or crashes, the log records in the log disk that have not been written to the backend data disk, that is, the records marked as commit status, need to be replayed and written to the backend storage system data disk. See Figure 4As shown, this solves the existing problem that "traditional NFS systems have difficulties in ensuring data consistency and integrity after a system crash or unexpected restart."

[0099] The method provided in the above embodiment utilizes the persistence characteristics of the NVMe log disk to ensure that all write requests are reliably recorded. After the system restarts or crashes, check whether there is data that needs to be recovered in the NVMe log disk, that is, the log record marked as commit. If there is, the data is played back, and if not, no playback is required. Through this mechanism, the consistency and integrity of the data can be guaranteed. And this mechanism does not require human intervention, and is automatically completed when the system starts. Compared with the data comparison method, the use of the log playback mechanism for fault recovery is simple and fast, thereby simplifying the fault recovery process and meeting the high requirements of enterprise-level storage systems for data reliability.

[0100] The method provided by the embodiment of the present invention has at least the following beneficial effects compared with the prior art:

[0101] 1. Introduce cache logs so that write requests can be recorded quickly. This cache log has the characteristics of high read and write performance and low latency, which can significantly improve the write performance of the NFS gateway and reduce real-time write pressure, thereby significantly improving the write performance and data reliability of the NFS gateway in high-concurrency scenarios.

[0102] 2. Introduce an asynchronous write mechanism to record the write request to the NVMe log disk first, and then asynchronously write the data to the backend storage system. This mechanism allows the write operation to not wait for the response of the backend storage system, which not only improves the concurrent processing capability of the system, but also reduces the blocking of the front-end NFS service by the write operation, thereby improving the overall system performance and reducing the write delay.

[0103] 3. Introduce a log cleaning and management mechanism. After confirming that the write request is successfully written to the back-end storage system, clean up the corresponding log records in time to ensure that the space of the log disk will not be occupied by invalid logs, maintain the efficient operation of the log disk, and further improve the stability and performance of the system.

[0104] 4. Introduce a fault recovery mechanism to determine whether playback is needed based on the status of log records. If playback is needed, write the data to the backend storage system again. This design not only improves the reliability of the system, but also simplifies the fault recovery process, ensures data consistency, and meets the high requirements of enterprise-level storage systems for data reliability.

[0105] 5. Optimize the balance between hardware cost and performance. We selected the cost-effective NVMe SSD as the log disk. Compared with more expensive storage media, NVMe SSD achieves a good balance between performance and cost. Through reasonable hardware selection, this solution not only achieves high-performance storage, but also controls the hardware cost of the system, and has high practical application value.

[0106] See also Figure 5 , shows a schematic diagram of main modules of a data processing device 500 provided in an embodiment of the present invention, which is applied to a distributed file system protocol NFS server, including:

[0107] The communication module 501 is used to receive a data processing request transmitted by the NFS client and send the data processing request to the protocol layer;

[0108] The protocol module 502 is used to parse the data processing request to obtain the request type and request parameters. When the request type is a write request, the request parameters are written into the log disk of the log layer in the form of logs to obtain log records.

[0109] The log processing module 503 is used to return the NFS client response success information through the communication layer after the writing is successful; and asynchronously read the log record from the log disk and send it to the interface layer;

[0110] The interface driver module 504 is used to determine the storage location in the backend storage system through the interface layer according to the request parameters in the log record, and call the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0111] In the implementation device of the present invention, the data processing request also includes NFS protocol version information, and the corresponding processing service is determined according to the NFS protocol version information.

[0112] In the implementation device of the present invention, the request parameters include file name, offset, length, and the interface driver module 504 is used to:

[0113] Determine the corresponding storage node from the backend storage system according to the file name;

[0114] According to the offset and length, determine the corresponding storage location from the storage node;

[0115] A driving mode corresponding to the storage node in the driving layer is called to write the data to be written and the file attribute into the storage location of the storage node.

[0116] The implementation device of the present invention also includes a state module, which is used to:

[0117] Marking the state of the log record written to the log disk as a committed state;

[0118] After the calling interface layer and the driver layer write the data to be written in the log record into the backend storage system, the state of the log record is updated from the submitted state to the applied state;

[0119] Triggers the clearing operation of log records in the log disk whose status is application status.

[0120] The implementation device of the present invention also includes a recovery module, which is used to:

[0121] In response to the NFS server failure restart operation, check whether there is a log record in the log layer with a status of commit;

[0122] In response to the check result being existence, the interface layer and the driver layer are called to write the to-be-written data in the log record into the backend storage system.

[0123] In the implementation device of the present invention, the NFS server further includes a data cache layer, and the device further includes a reading module for:

[0124] When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cache data is queried from the data cache layer according to the file name, offset, and length;

[0125] In response to cache data misses, the corresponding storage node and storage location are determined from the backend storage system according to the file name, offset and length, and then the data is read from the storage location of the storage node and returned to the client.

[0126] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0127] Figure 6 An exemplary system architecture 600 to which embodiments of the present invention may be applied is shown, including terminal devices 601 , 602 , 603 , a network 604 and a server 605 (only an example).

[0128] Terminal devices 601, 602, 603 can be various electronic devices with display screens and supporting web browsing, and various communication client applications are installed. Users can use terminal devices 601, 602, 603 to interact with server 605 through network 604 to receive or send messages, etc.

[0129] The network 604 is used to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0130] The server 605 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 501, 502, and 503 (for example only). The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 605, and accordingly, the device is generally set in the server 605.

[0131] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0132] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0133] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0134] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0135] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0136] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a communication module, a protocol module, a log processing module, and an interface driver module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, an interface driver module may also be described as an "interface module and a driver module."

[0139] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes any of the above data processing methods.

[0140] The computer program product of the present invention comprises a computer program, and when the computer program is executed by a processor, the data processing method in the embodiment of the present invention is implemented.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, It is characterized in that Applied to the distributed file system protocol NFS server, the NFS server includes the communication layer, protocol layer, log layer, interface layer and driver layer, including: Receive a data processing request transmitted by the NFS client through the communication layer, and send the data processing request to the protocol layer; Parse the data processing request through the protocol layer to obtain the request type and request parameters. If the request type is a write request, write the request parameters into the log disk of the log layer in the form of logs to obtain log records. After the writing is successful, the NFS client is returned a response success message through the communication layer; and the log processing module of the log layer asynchronously reads the log record from the log disk and sends it to the interface layer; The interface layer determines the storage location in the backend storage system according to the request parameters in the log record, and calls the driver mode corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

2. The method according to claim 1, It is characterized in that The data processing request also includes NFS protocol version information, and the corresponding processing service is determined according to the NFS protocol version information.

3. The method according to claim 1, It is characterized in that The request parameters include a file name, an offset, a length, the data to be written, and file attributes. The storage location is determined in the backend storage system according to the request parameters in the log record, and a driver mode corresponding to the storage location is called in the driver layer to write the data to be written in the request parameters into the storage location, including: Determine the corresponding storage node from the backend storage system according to the file name; According to the offset and length, determine the corresponding storage location from the storage node; A driving mode corresponding to the storage node in the driving layer is called to write the data to be written and the file attribute into the storage location of the storage node.

4. The method according to claim 1, It is characterized in that The method further comprises: Marking the state of the log record written to the log disk as a committed state; After the calling interface layer and the driver layer write the data to be written in the log record into the backend storage system, the state of the log record is updated from the submitted state to the applied state; Triggers the clearing operation of log records in the log disk whose status is application status.

5. The method according to claim 4, It is characterized in that The method further comprises: In response to the NFS server failure restart operation, check whether there is a log record in the log layer with a status of commit; In response to the check result being existence, the interface layer and the driver layer are called to write the to-be-written data in the log record into the backend storage system.

6. The method according to claim 1, It is characterized in that The NFS server also includes a data cache layer, and the method further includes: When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cache data is queried from the data cache layer according to the file name, offset, and length; In response to cache data misses, the corresponding storage node and storage location are determined from the backend storage system according to the file name, offset and length, and then the data is read from the storage location of the storage node and returned to the client.

7. A data processing device, It is characterized in that Applicable to distributed file system protocol NFS server, including: The communication module is used to receive a data processing request transmitted by the NFS client and send the data processing request to the protocol layer; A protocol module, used to parse the data processing request, obtain the request type and request parameters, and when the request type is a write request, write the request parameters into a log disk of the log layer in the form of a log to obtain a log record; The log processing module is used to return the NFS client response success information through the communication layer after the writing is successful; and asynchronously read the log record from the log disk and send it to the interface layer; The interface driver module is used to determine the storage location in the backend storage system through the interface layer according to the request parameters in the log record, and call the driving method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

8. The device according to claim 7, It is characterized in that The request parameters include file name, offset, length, and the interface driver module, which is used to: Determine the corresponding storage node from the backend storage system according to the file name; According to the offset and length, determine the corresponding storage location from the storage node; A driving mode corresponding to the storage node in the driving layer is called to write the data to be written and the file attribute into the storage location of the storage node.

9. The device according to claim 7, It is characterized in that The device also includes a state module, configured to: Marking the state of the log record written to the log disk as a committed state; After the calling interface layer and the driver layer write the data to be written in the log record into the backend storage system, the state of the log record is updated from the submitted state to the applied state; Triggers the clearing operation of log records in the log disk whose status is application status.

10. An electronic device, It is characterized in that include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

11. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

12. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Working method of high-reliability distributed log system

    CN108108476A

  • Cloud computing node and file management method and device

    CN113032099A

  • Log processing method, device and system, chip, equipment and storage medium

    CN113051243A

  • Metadata processing method, device and equipment and computer storage medium

    CN115203156A

  • Multi-level read caching for multiplexed transactional logging

    US20080133615A1