A data processing method and apparatus

Abstract

This invention discloses a data processing method and apparatus, relating to the field of data processing technology in the big data domain. One specific implementation of the method includes: receiving a data processing request transmitted by an NFS client through a communication layer and sending it to a protocol layer; parsing the request at the protocol layer to obtain the request type and request parameters, wherein the request type is a write request; writing the request parameters to a log disk in the log layer in log form to obtain a log record; returning a success response to the NFS client through the communication layer after successful writing; asynchronously reading the log record from the log disk through the log processing module of the log layer and sending it to the interface layer; determining the storage location in the backend storage system based on the request parameters through the interface layer, and calling the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters to the storage location. This implementation features high read / write performance and low latency, significantly improving the write performance of the NFS gateway.

Description

Technical Field

[0001] This invention relates to the field of data processing technology in the big data field, and in particular to a data processing method and apparatus. Background Technology

[0002] With the rapid development of big data and cloud computing, distributed file systems are playing an increasingly important role in enterprise-level storage solutions. NFS (Network File System), as a mature and widely used distributed file system protocol, is widely used in various application scenarios. However, traditional NFS servers often face performance bottlenecks and data consistency issues in high-concurrency, high-data-volume scenarios.

[0003] To improve the performance and reliability of NFS servers, NFS gateways, acting as user-space NFS servers, offer flexible architectures and rich functional support. However, in implementing this invention, the inventors discovered at least the following problems in the prior art: under high-concurrency write scenarios, NFS gateways may still face issues with write latency and data consistency. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data processing method and apparatus that can at least solve the problems of write latency and data consistency faced by the prior art in high-concurrency write scenarios.

[0005] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, applied to a distributed file system protocol NFS server. The NFS server includes a communication layer, a protocol layer, a log layer, an interface layer, and a driver layer, comprising:

[0006] The data processing request is received from the NFS client through the communication layer and then sent to the protocol layer.

[0007] The data processing request is parsed by the protocol layer to obtain the request type and request parameters. In the case of a write request, the request parameters are written to the log disk of the log layer in the form of logs to obtain log records.

[0008] After successful writing, a success message is returned to the NFS client via the communication layer; and the log record is asynchronously read from the log disk and sent to the interface layer via the log processing module of the log layer.

[0009] The interface layer determines the storage location in the backend storage system based on the request parameters in the log records, and calls the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0010] Optionally, the data processing request may also include NFS protocol version information, and the corresponding processing service may be determined based on the NFS protocol version information.

[0011] Optionally, the request parameters include filename, offset, length, the data to be written, and file attributes. The step of determining the storage location in the backend storage system based on the request parameters in the log record, and calling the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters to the storage location, includes:

[0012] The corresponding storage node is determined from the backend storage system based on the filename;

[0013] The corresponding storage location is determined from the storage node based on the offset and length;

[0014] The driver method corresponding to the storage node in the driver layer is invoked to write the data to be written and the file attributes into the storage location of the storage node.

[0015] Optionally, the method further includes:

[0016] Mark the status of the log record written to the log disk as committed;

[0017] After the interface layer and driver layer write the data to be written in the log record to the backend storage system, the status of the log record is updated from the commit status to the application status.

[0018] Trigger a clear operation on log records in the log disk that are in the application state.

[0019] Optionally, the method further includes:

[0020] In response to an NFS server failure restart, check the log layer for log records with a committed status.

[0021] In response to the check result indicating existence, the interface layer and driver layer are invoked to write the data to be written in the log record to the backend storage system.

[0022] Optionally, the NFS server further includes a data caching layer, and the method further includes:

[0023] When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cached data is queried from the data cache layer based on the file name, offset, and length.

[0024] In response to a cache miss, the system determines the appropriate storage node and storage location from the backend storage system based on the filename, offset, and length, and then reads the data from the storage location of the storage node and returns it to the client.

[0025] To achieve the above objectives, according to another aspect of the present invention, a data processing apparatus is provided, applied to a distributed file system protocol NFS server, comprising:

[0026] The communication module is used to receive data processing requests transmitted by the NFS client and send the data processing requests to the protocol layer;

[0027] The protocol module is used to parse the data processing request to obtain the request type and request parameters. In the case of a write request, the request parameters are written to the log disk of the log layer in the form of logs to obtain log records.

[0028] The log processing module is used to return a success response message to the NFS client through the communication layer after a successful write; and to asynchronously read the log record from the log disk and send it to the interface layer;

[0029] The interface driver module is used to determine the storage location in the backend storage system based on the request parameters in the log records through the interface layer, and call the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0030] Optionally, the data processing request may also include NFS protocol version information, and the corresponding processing service may be determined based on the NFS protocol version information.

[0031] Optionally, the request parameters include filename, offset, length, and the interface driver module, used for:

[0032] The corresponding storage node is determined from the backend storage system based on the filename;

[0033] The corresponding storage location is determined from the storage node based on the offset and length;

[0034] The driver method corresponding to the storage node in the driver layer is invoked to write the data to be written and the file attributes into the storage location of the storage node.

[0035] Optionally, the device further includes a status module for:

[0036] Mark the status of the log record written to the log disk as committed;

[0037] After the interface layer and driver layer write the data to be written in the log record to the backend storage system, the status of the log record is updated from the commit status to the application status.

[0038] Trigger a clear operation on log records in the log disk that are in the application state.

[0039] Optionally, the device further includes a recovery module for:

[0040] In response to an NFS server failure restart, check the log layer for log records with a committed status.

[0041] In response to the check result indicating existence, the interface layer and driver layer are invoked to write the data to be written in the log record to the backend storage system.

[0042] Optionally, the NFS server further includes a data caching layer, and the apparatus further includes a read module for:

[0043] When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cached data is queried from the data cache layer based on the file name, offset, and length.

[0044] In response to a cache miss, the system determines the appropriate storage node and storage location from the backend storage system based on the filename, offset, and length, and then reads the data from the storage location of the storage node and returns it to the client.

[0045] To achieve the above objectives, according to another aspect of the present invention, a data processing electronic device is provided.

[0046] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the data processing methods described above.

[0047] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described above.

[0048] To achieve the above objectives, according to another aspect of the present invention, a computing program product is provided. One computing program product of the present invention includes a computer program, which, when executed by a processor, implements the data processing method provided in the present invention.

[0049] According to the solution provided by the present invention, one embodiment of the invention has the following advantages or beneficial effects: A high-speed cache log is introduced as an intermediate cache layer for quickly recording write request logs. Upon successful writing to the log disk, a success response is immediately returned to the client. The subsequent writing process to the backend storage system is asynchronous. This asynchronous operation shortens the I / O path, reduces I / O latency, improves write performance, and solves the existing write latency problem. Furthermore, since the write is to the log disk, there is no pollution of backend data, thus solving the existing data inconsistency problem. Because only the log disk uses NVMe, the backend data disk can use a regular mechanical disk; the combination of the two fully utilizes hardware characteristics and reduces hardware costs. This high-speed cache log of the present application features high read / write performance and low latency, which can significantly improve the write performance of the NFS gateway.

[0050] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0051] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0052] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the NFS gateway structure according to an embodiment of the present invention;

[0054] Figure 3 This is a flowchart illustrating an optional data processing method according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic flowchart of a specific data processing method according to an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the main modules of a data processing device according to an embodiment of the present invention;

[0057] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0058] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0059] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0060] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0061] Where there is no conflict, the embodiments and features in the embodiments of this invention can be combined with each other. The acquisition, transmission, storage, use, and processing of data in the technical solutions of this invention comply with the relevant provisions of national laws and regulations, are used for legal and reasonable purposes, and are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities.

[0062] Regarding user information, necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access such data comply with relevant laws and regulations, and safeguard the security of user personal information. Once this user personal information data is no longer needed, risks should be minimized by restricting or even prohibiting data collection and / or deleting the data. Where applicable, including in certain relevant applications, user privacy should be protected through data de-identification, such as by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the specific address level), controlling how data is stored, and / or other de-identification methods.

[0063] The shortcomings of the existing technology are described in detail here:

[0064] 1. Write Performance Bottlenecks: 1) High Write Latency: Traditional NFS typically waits for data to be written to the backend storage system during write operations, resulting in high write latency. This latency increases significantly, especially in high-concurrency environments, impacting overall system performance. 2) Blocking Caused by Synchronous Writes: Many traditional NFS implementations use synchronous writes, where each write request waits for the write operation to complete before returning a completion message to the client. This synchronous write mode easily leads to request blocking, affecting system response time and concurrency processing capabilities.

[0065] 2. Data Consistency Issues. 1) Difficult Crash Recovery: Traditional NFS systems struggle to ensure data consistency and integrity after a system crash or unexpected restart. The lack of an effective logging mechanism can lead to data loss or inconsistency. 2) Weak Consistency Model: The weak consistency model of traditional NFS (such as NFSv3's asynchronous write mode) may not meet strict data consistency requirements in certain application scenarios, posing a challenge to applications with high data consistency requirements (such as database applications).

[0066] 3. Insufficient utilization of hardware resources. High cost and low efficiency: Traditional NFS systems have problems with hardware resource utilization, especially under high-performance storage requirements, resulting in high hardware costs and low efficiency. For application scenarios requiring high IOPS (input / output operations per second) and low latency, traditional hard drives may not be able to meet the requirements.

[0067] See Figure 1 The diagram shows the main flowchart of a data processing method provided by an embodiment of the present invention, which includes the following steps:

[0068] S101: Receive data processing requests transmitted by NFS clients through the communication layer and send the data processing requests to the protocol layer;

[0069] S102: The data processing request is parsed by the protocol layer to obtain the request type and request parameters. If the request type is a write request, the request parameters are written to the log disk of the log layer in the form of logs to obtain log records.

[0070] S103: After successful writing, return a success message to the NFS client through the communication layer; and asynchronously read the log record from the log disk and send it to the interface layer through the log processing module of the log layer;

[0071] S104: Based on the request parameters in the log records, the interface layer determines the storage location in the backend storage system and calls the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0072] In the above embodiments, an NFS gateway is a device used to connect different network storage systems. Its main function is to convert the NFS protocol into other protocols, such as CephFS or HDFS, so that different storage systems can provide services via NFS. An NFS gateway enables transparent cross-platform access to file systems, improving the flexibility and scalability of data sharing.

[0073] For steps S101 to S104, the NFS gateway (i.e., the NFS server) in this solution is responsible for receiving and parsing NFS client I / O requests, recording the I / O requests in the NFS protocol into the log layer, and returning a success response message to the NFS client after the log record is successfully written. The log processing module of the log layer asynchronously reads the log records and sends them to the Backend interface layer for processing. The NFS gateway consists of an RPC (Remote Procedure Call) communication layer, an NFS protocol layer, a data cache and log layer, a Backend interface layer, and a driver layer. See [link to relevant documentation]. Figure 2 As shown.

[0074] 1. RPC Communication Layer. The NFS client initiates I / O requests to the NFS gateway, including file read and write operations. The RPC communication layer is responsible for receiving I / O requests from the NFS client. Since NFS communication uses the RPC protocol, communication with the client requires the support of the RPC communication layer.

[0075] 2. NFS Protocol Layer. The RPC communication layer forwards I / O requests to the NFS protocol layer for processing. The NFS protocol layer parses the I / O request, obtaining the request type, NFS protocol version information, and request parameters. The request type is either a read request or a write request. A read request means the client wants to read file data from the server, while a write request means it wants to write data to a file on the server. During the parsing process, the NFS protocol layer also identifies the NFS protocol version used by the client request. Different versions of the NFS protocol differ in functionality, security, and performance. For example, NFSv4 introduces a stronger security model (such as Kerberos support) and adds Compound operations to reduce network round trips compared to NFSv3. Based on the NFS protocol version information, the NFS server decides whether to use NFSv3 or NFSv4 to handle the request. This choice is usually determined by both the server configuration and the capabilities of the client request. If both parties support and are configured to use NFSv4, NFSv4 will be selected to provide more advanced features; otherwise, if the client or server only supports NFSv3, or is configured to prioritize NFSv3, NFSv3 will be used to process requests.

[0076] When the request is a read request, the system queries the data cache for data matching the request. If the cache is hit, the cached data is returned. However, if the cache is not hit, the system queries the backend storage system for data matching the request and returns it. When the request is a write request, the NFS protocol layer writes the request information to the NVMe log disk in the log layer as a log. This step ensures fast persistence of write requests and reduces the risk of data loss due to network or system failures. The NVMe log disk refers to a high-speed solid-state storage disk using the NVMe interface standard, commonly used for storing system logs, database transaction logs, and other applications sensitive to write speed and latency.

[0077] 3. Data caching and logging layer. In the NFS protocol, I / O requests (whether read or write) typically contain the following key information, which is crucial for locating and processing file data:

[0078] 1) File Name: A unique name that identifies the remote file that the client wants to access. The file name helps the server locate the specific file resource.

[0079] 2) Offset: Specifies the starting position of a read or write operation within the file. For a read request, it tells the server from which byte in the file to begin reading data; for a write request, it indicates where in the file the data should be written. The offset ensures precise positioning of data operations.

[0080] 3) Length: Specifies the size of the data block to be read or written. In a read request, it defines how much data should be read starting from the specified offset; in a write request, it specifies the amount of data to be written. This helps control the amount of data transferred in each I / O operation.

[0081] 4) Data: In a write request, the actual data to be written to the file is included in the request. This data is sent to the server along with the request so that the server can write it to the specified location in the file.

[0082] 5) File Attributes: While not included in every I / O request, in certain situations (especially metadata update operations), the request may carry information about file attributes, such as modification time, permissions, and ownership. This information is used to update the file's status or verify the legitimacy of the operation.

[0083] The log layer stores I / O request logs received by the NFS gateway, providing high-performance write and read capabilities. By combining this information, the NFS gateway can accurately execute client read and write requests, ensuring correct data processing and storage. After a successful write to the log, the NFS gateway returns a success response message to the NFS client, which can be achieved through the RPC communication layer. This approach solves the problem that many traditional NFS implementations use synchronous write methods, where each write request waits for the write operation to complete before returning a completion message to the client.

[0084] Subsequently, the log processing module is notified, and the system asynchronously reads log records from the NVMe log disk and sends them to the Backend interface layer. The Backend interface layer then writes the actual data to the backend storage system. This asynchronous operation effectively reduces the real-time write pressure on the NFS service, improves the overall system's concurrent processing capabilities, and solves the problem that "traditional NFS typically needs to wait for data to be written to the backend storage system, which leads to high write latency." After the data is successfully written to the backend storage system, the corresponding log records can be deleted from the NVMe log disk to free up storage space.

[0085] 4. The Backend interface layer and driver layer are responsible for translating the NFS I / O protocol into requests that the specific backend storage system can accept, and then sending them to the relevant distributed file system cluster through the Backend driver. I / O requests initiated by NFS clients are in the standard NFS protocol format. The Backend interface layer first receives log records through the log interface layer and converts these NFS protocol format requests into a format that the backend storage system can understand. This is because backend storage systems (such as HDFS, Ceph, GlusterFS, etc.) typically have their own communication protocols and APIs (Application Programming Interfaces), which are not directly compatible with the NFS protocol. The conversion is not only at the protocol level but also includes adapting and encapsulating request parameters. For example, information such as file offsets and lengths in the NFS request are mapped to corresponding parameters in the backend storage system to ensure that the content and intent of the request are correctly conveyed.

[0086] In an NFS server architecture, the "Backend interface layer" acts as a bridge. Backend storage systems typically consist of multiple nodes, each responsible for storing a portion of the data. The Backend interface layer also needs to determine which specific storage node and storage location to route the request to based on the filename, offset, and length. To interact with different storage nodes, the Backend interface layer relies on specialized Backend drivers, such as CephfsBackend, VFSBackend, GlusterBackend, and NVMe drivers. These drivers are software modules specific to each storage node, providing the interface and logic for interacting with that system. The Backend interface layer sends the transformed and encapsulated data to be written and file attributes to the specific storage node by calling the appropriate driver. The Backend interface layer ensures the transparency of the NFS protocol, allowing NFS clients to seamlessly interact with the diverse underlying distributed file storage systems without needing to directly understand their complex details.

[0087] The method provided in the above embodiments introduces a high-speed cache log, namely an NVMe log disk, as an intermediate cache layer for quickly recording write request logs. Upon successful writing to the log disk, a success response is immediately returned to the client. Subsequent data writing to the backend storage system is asynchronous. This asynchronous operation shortens the I / O path, reduces I / O latency, improves write performance, and solves the existing write latency problem. Furthermore, since the write is to the log disk, there is no pollution of backend data, resolving the data inconsistency problem. Because only the log disk uses NVMe, the backend data disk can use a regular mechanical disk; the combination of the two fully utilizes hardware characteristics and reduces hardware costs. This high-speed cache log of this application features high read / write performance and low latency, significantly improving the write performance of the NFS gateway.

[0088] See Figure 3 The diagram illustrates an optional data processing method according to an embodiment of the present invention, including the following steps:

[0089] S301: Receive data processing requests transmitted by NFS clients through the communication layer and send the data processing requests to the protocol layer;

[0090] S302: Parse the data processing request through the protocol layer to obtain the request type and request parameters. If the request type is a write request, write the request parameters to the log disk of the log layer in the form of a log to obtain a log record. Mark the status of the log record as a committed status.

[0091] S303: After successful writing, return a success message to the NFS client through the communication layer; and asynchronously read the log record from the log disk and send it to the interface layer through the log processing module of the log layer;

[0092] S304: Based on the request parameters in the log records, the interface layer determines the storage location in the backend storage system and calls the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters to the storage location.

[0093] S305: Update the status of the log record in the log layer from the commit status to the application status, and trigger the clearing operation of the log record in the log disk that is in the application status;

[0094] S306: In response to an NFS server failure restart, check the log layer for log records with a committed status.

[0095] S307: In response to the check result being present, the interface layer and driver layer are invoked to write the data to be written in the log record to the backend storage system.

[0096] In the above implementation, for steps S301 to S307, during the entire I / O request processing, the NFS client write process to the NVMe log disk is a synchronous call, and the NVMe log disk write process to the backend storage system data disk is an asynchronous call.

[0097] Upon successful writing to the NVMe log disk, log records successfully written by the client are marked as committed. The log processing module periodically reads committed log records from the NVMe log disk, writes the data in these records to the backend data disk, and changes the status of the corresponding log record in the log disk to applied. Records are not deleted from the log disk at this point. They are only deleted when the log disk capacity threshold is reached, for example, when the capacity is 100 log records.

[0098] In cases of NVMe log disk write failures, when a client fails to write to the NVMe log disk, the failure record is not written to the NVMe log disk. The client will receive a write failure message and should re-initiate the write operation. Therefore, even if the system crashes, the impact only applies to the write operation from the NVMe log disk to the backend storage system data disk. Since client write operations are recorded in the NVMe log disk but not the backend data disk, after a system restart or crash, it is necessary to replay the log records in the log disk that have not been written to the backend data disk—that is, the records marked as committed—and write them to the backend storage system data disk. See [link to relevant documentation]. Figure 4As shown, this solves the problem that "traditional NFS systems have difficulty ensuring data consistency and integrity after system crashes or unexpected restarts".

[0099] The method provided in the above embodiments utilizes the persistence characteristics of NVMe log disks to ensure that all write requests are reliably recorded. After a system restart or crash, it checks whether there is data in the NVMe log disk that needs to be recovered, i.e., log records marked as committed. If so, data replay is performed; otherwise, replay is not required. This mechanism ensures data consistency and integrity. Furthermore, this mechanism requires no human intervention and is completed automatically during system startup. Compared to data comparison methods, using log replay for fault recovery is simpler and faster, thus simplifying the fault recovery process and meeting the high data reliability requirements of enterprise-level storage systems.

[0100] The method provided in this invention has at least the following advantages compared to the prior art:

[0101] 1. Introducing high-speed cached logs allows write requests to be recorded quickly. This cached log features high read / write performance and low latency, significantly improving the write performance of the NFS gateway, reducing real-time write pressure, and thus significantly enhancing the write performance and data reliability of the NFS gateway in high-concurrency scenarios.

[0102] 2. An asynchronous write mechanism is introduced, where write requests are first recorded on the NVMe log disk, and then the data is asynchronously written to the backend storage system. This mechanism eliminates the need for write operations to wait for a response from the backend storage system, improving the system's concurrency capabilities and reducing the blocking of write operations on the frontend NFS service, thereby enhancing overall system performance and reducing write latency.

[0103] 3. A log cleanup and management mechanism was introduced. After confirming that the write request was successfully written to the backend storage system, the corresponding log records were cleaned up in a timely manner. This ensured that the space of the log disk was not occupied by invalid logs, maintained the efficient operation of the log disk, and further improved the stability and performance of the system.

[0104] 4. Introduce a fault recovery mechanism. Determine whether replay is needed based on the status recorded in the logs. If replay is needed, the data writing operation to the backend storage system is executed again. This design not only improves the reliability of the system, but also simplifies the fault recovery process, ensures data consistency, and meets the high requirements of enterprise-level storage systems for data reliability.

[0105] 5. Optimizing the balance between hardware cost and performance. A cost-effective NVMe SSD was selected as the journal drive. Compared to more expensive storage media, NVMe SSDs achieve a good balance between performance and cost. Through reasonable hardware selection, this solution not only achieves high-performance storage but also controls system hardware costs, demonstrating high practical application value.

[0106] See Figure 5 This diagram illustrates the main modules of a data processing device 500 provided in an embodiment of the present invention, applied to a distributed file system protocol NFS server, including:

[0107] Communication module 501 is used to receive data processing requests transmitted by NFS clients and send the data processing requests to the protocol layer;

[0108] Protocol module 502 is used to parse the data processing request to obtain the request type and request parameters. When the request type is a write request, the request parameters are written to the log disk of the log layer in the form of logs to obtain log records.

[0109] Log processing module 503 is used to return a success response message to the NFS client through the communication layer after a successful write; and to asynchronously read the log record from the log disk and send it to the interface layer;

[0110] The interface driver module 504 is used to determine the storage location in the backend storage system based on the request parameters in the log records through the interface layer, and call the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

[0111] In the apparatus of this invention, the data processing request further includes NFS protocol version information, and the corresponding processing service is determined based on the NFS protocol version information.

[0112] In the apparatus of this invention, the request parameters include filename, offset, and length. The interface driver module 504 is used for:

[0113] The corresponding storage node is determined from the backend storage system based on the filename;

[0114] The corresponding storage location is determined from the storage node based on the offset and length;

[0115] The driver method corresponding to the storage node in the driver layer is invoked to write the data to be written and the file attributes into the storage location of the storage node.

[0116] The apparatus for implementing the present invention further includes a status module, used for:

[0117] Mark the status of the log record written to the log disk as committed;

[0118] After the interface layer and driver layer write the data to be written in the log record to the backend storage system, the status of the log record is updated from the commit status to the application status.

[0119] Trigger a clear operation on log records in the log disk that are in the application state.

[0120] The apparatus of the present invention further includes a recovery module, used for:

[0121] In response to an NFS server failure restart, check the log layer for log records with a committed status.

[0122] In response to the check result indicating existence, the interface layer and driver layer are invoked to write the data to be written in the log record to the backend storage system.

[0123] In the apparatus of this invention, the NFS server further includes a data caching layer, and the apparatus further includes a read module for:

[0124] When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cached data is queried from the data cache layer based on the file name, offset, and length.

[0125] In response to a cache miss, the system determines the appropriate storage node and storage location from the backend storage system based on the filename, offset, and length, and then reads the data from the storage location of the storage node and returns it to the client.

[0126] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0127] Figure 6 An exemplary system architecture 600 to which embodiments of the present invention can be applied is shown, including terminal devices 601, 602, 603, network 604, and server 605 (only an example).

[0128] Terminal devices 601, 602, and 603 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc.

[0129] Network 604 is a medium used to provide a communication link between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0130] Server 605 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 501, 502, and 503 (this is just an example). The backend management server can analyze and process received data such as product information query requests, and feed back the processing results (such as target push information and product information—this is just an example) to the terminal devices. It should be noted that the method provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the apparatus is generally set in server 605.

[0131] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0132] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0133] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0134] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0135] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0136] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor; for example, a processor can be described as including a communication module, a protocol module, a log processing module, and an interface driver module. The names of these modules do not necessarily limit the module itself; for example, the interface driver module can also be described as "interface module and driver module".

[0139] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the data processing methods described above.

[0140] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the data processing method in the embodiments of the present invention.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, This is applied to the NFS (Network File System) server, a distributed file system protocol. The NFS server comprises a communication layer, protocol layer, log layer, interface layer, and driver layer, including: The data processing request is received from the NFS client through the communication layer and then sent to the protocol layer. The data processing request is parsed by the protocol layer to obtain the request type and request parameters. In the case of a write request, the request parameters are written to the log disk of the log layer in the form of logs to obtain log records. After successful writing, a success message is returned to the NFS client via the communication layer; and the log record is asynchronously read from the log disk and sent to the interface layer via the log processing module of the log layer. The interface layer determines the storage location in the backend storage system based on the request parameters in the log records, and calls the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

2. The method according to claim 1, characterized in that, The data processing request also includes NFS protocol version information, and the corresponding processing service is determined based on the NFS protocol version information.

3. The method according to claim 1, characterized in that, The request parameters include filename, offset, length, the data to be written, and file attributes. The step of determining the storage location in the backend storage system based on the request parameters in the log record, and calling the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters to the storage location, includes: The corresponding storage node is determined from the backend storage system based on the filename; The corresponding storage location is determined from the storage node based on the offset and length; The driver method corresponding to the storage node in the driver layer is invoked to write the data to be written and the file attributes into the storage location of the storage node.

4. The method according to claim 1, characterized in that, The method further includes: Mark the status of the log record written to the log disk as committed; After the interface layer and driver layer write the data to be written in the log record to the backend storage system, the status of the log record is updated from the commit status to the application status. Trigger a clear operation on log records in the log disk that are in the application state.

5. The method according to claim 4, characterized in that, The method further includes: In response to an NFS server failure restart, check the log layer for log records with a committed status. In response to the check result indicating existence, the interface layer and driver layer are invoked to write the data to be written in the log record to the backend storage system.

6. The method according to claim 1, characterized in that, The NFS server also includes a data caching layer, and the method further includes: When the request type is a read request, the file name, offset, and length are obtained from the request parameters, and the corresponding cached data is queried from the data cache layer based on the file name, offset, and length. In response to a cache miss, the system determines the appropriate storage node and storage location from the backend storage system based on the filename, offset, and length, and then reads the data from the storage location of the storage node and returns it to the client.

7. A data processing apparatus, characterized in that, Applications to the NFS (Network File System) server, including: The communication module is used to receive data processing requests transmitted by the NFS client and send the data processing requests to the protocol layer; The protocol module is used to parse the data processing request to obtain the request type and request parameters. In the case of a write request, the request parameters are written to the log disk of the log layer in the form of logs to obtain log records. The log processing module is used to return a success response message to the NFS client through the communication layer after a successful write; and to asynchronously read the log record from the log disk and send it to the interface layer; The interface driver module is used to determine the storage location in the backend storage system based on the request parameters in the log records through the interface layer, and call the driver method corresponding to the storage location in the driver layer to write the data to be written in the request parameters into the storage location.

8. The apparatus according to claim 7, characterized in that, The request parameters include filename, offset, length, and the interface driver module, which is used for: The corresponding storage node is determined from the backend storage system based on the filename; The corresponding storage location is determined from the storage node based on the offset and length; The driver method corresponding to the storage node in the driver layer is invoked to write the data to be written and the file attributes into the storage location of the storage node.

9. The apparatus according to claim 7, characterized in that, The device further includes a status module for: Mark the status of the log record written to the log disk as committed; After the interface layer and driver layer write the data to be written in the log record to the backend storage system, the status of the log record is updated from the commit status to the application status. Trigger a clear operation on log records in the log disk that are in the application state.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images