System, method and apparatus for processing files

Abstract

The application discloses a file processing system, a file processing method and device. The system comprises a client, wherein an application program is deployed in the client, the application program initiates a request for file data processing to a key-value pair system and / or a persistent system through the client to provide services to the outside world, the key-value pair system, and a persistent file system, wherein the persistent file system is used for persistently storing file data. The application program interacts with the client through a portable operating system interface, the client interacts with the key-value pair system through an interface adapted to the key-value pair system, and the client interacts with the persistent file system through the portable operating system interface. The application solves the technical problem that in the prior art, the source code of the application program needs to be modified and adapted for using the key-value pair system, resulting in a relatively high error rate during data processing.

Description

Technical Field

[0001] This application relates to the field of document data processing technology, and more specifically, to a document processing system, document processing method, and apparatus. Background Technology

[0002] AI training scenarios are characterized by high concurrency, frequent read operations, and a massive number of small files. Traditional distributed file processing systems are ill-suited to the I / O characteristics of AI training scenarios. On the other hand, traditional key-value pair systems, especially memory-centric ones, offer simple interfaces, high performance, and ease of maintenance, making them a good match for the I / O load characteristics of AI training scenarios. Therefore, key-value pair systems can serve as a cache between the application and the persistent file system, improving the application's file I / O throughput and reducing I / O latency. However, using a key-value pair system requires modifying the application's source code, adapting the POSIX API to the key-value pair system's get and put interfaces, which incurs significant manpower costs and has a relatively high error rate during data processing.

[0003] Existing distributed data caching layers, such as open-source distributed in-memory file processing systems, consist of a central master node, storage worker nodes, and the underlying file processing system, UFS (Used File System). The central master node centrally manages the cached file directory tree information, active and invalid worker node information, and the organizational relationships between all files and blocks, and blocks and workers. Before reading a file, a client needs to communicate with the central master node to obtain the file's block information, and then initiate a data read / write request with the storage worker node where the block content is located. The storage worker node manages the locally cached block information, responds to client requests to read block data, and handles requests to pull or persist dirty data from UFS. The central master node in open-source distributed in-memory file processing systems limits scalability and availability. Clients need to perform multiple RPC communications with the central master node and storage worker nodes to perform file data I / O. When facing high-concurrency reads of massive numbers of files, the central master node faces enormous query and metadata storage pressure. The latency from multiple RPC operations when reading small files significantly slows down the application's file access throughput.

[0004] To address the issue that the aforementioned technologies require modifications to the application's source code for use with key-value pairs, leading to a high error rate during data processing, no effective solution has yet been proposed. Summary of the Invention

[0005] This application provides a file processing system, file processing method, and apparatus to at least solve the technical problem in the related art that the source code of the application needs to be modified and adapted to use a key-value pair system, resulting in a relatively high error rate during data processing.

[0006] According to one aspect of the embodiments of this application, a file processing system is provided, comprising: a client, wherein an application is deployed in the client, the application initiating a file data processing request to a key-value pair system and / or a persistent file system through the client to provide services externally; the key-value pair system, wherein the key-value pair system serves as a cache database between the client and a persistent file system; and the persistent file system, wherein the persistent file system is used for persistently storing file data; wherein the application interacts with the client for file data through a portable operating system interface, the client interacts with the key-value pair system for file data through an interface adapted to the key-value pair system, and the client interacts with the persistent file system for file data through the portable operating system interface.

[0007] Furthermore, the files in the key-value pair system are stored in the form of multiple blocks of preset length, wherein the file name and the block number of the block content are used as keys and the block content is used as values ​​to form a first key-value pair, and the first key-value pair is stored in the key-value pair system.

[0008] Furthermore, the key-value pair system is also used to store file attribute information and symbolic links, wherein the file name and attribute instructions are used as keys, and the file attribute information and / or the file symbolic links are used as values ​​to form a second key-value pair, and the second key-value pair is stored in the key-value pair system.

[0009] Furthermore, the key-value pair system is also used to store the directory tree of the file, wherein the parent node name of the directory tree is used as the key and the operation log information of the parent node is used as the value to form a component as a third key-value pair, and the third key-value pair is stored in the key-value pair system.

[0010] According to another aspect of the embodiments of this application, a file processing method is also provided, comprising: the processing method being applied to the file processing system described above, comprising: acquiring a data processing request initiated by an application for file data; determining the request type of the data processing request, wherein the request type is one of the following: a read file request, a write file request, and a directory processing request; and performing data processing on the file data according to the request type.

[0011] Furthermore, if the data processing request is a file read request, the data processing of the file data according to the request type includes: determining the first block number based on the first file name, the first starting byte, and the length of the data to be read in the file read request; obtaining the first block content based on the first file name and the first block number; and feeding back the first block content to the application.

[0012] Further, obtaining the first block of content based on the first file name and the first block number, and feeding the first block of content back to the application includes: determining whether the client stores the first block of content based on the first file name and the first block number; if the client stores the first block of content, obtaining the first block of content from the client and feeding the first block of content back to the application; if the client does not store the first block of content, forwarding the file read request to the key-value pair system through the client, and determining whether the key-value pair system stores the first block of content based on the first file name and the first block number; if the key-value pair system stores the first block of content, caching the first block of content stored in the key-value pair system to the client, and feeding the first block of content back to the application through the client; if the key-value pair system does not store the first block of content, forwarding the file read request to the persistent file system through the client, obtaining the first block of content from the persistent file system, and feeding the first block of content back to the application through the client.

[0013] Furthermore, after the first block of content is fed back to the application through the client, the method further includes: storing the first block of content in the form of key-value pairs in the key-value pair system through the client.

[0014] Furthermore, if the type of the data processing request is a write file request, the data processing of the file data according to the request type includes: forwarding the write file request to the persistent file system through the client, completing the write file request in the persistent file system; and after the persistent file system completes the write file request, writing the target data information corresponding to the write file request into the key-value pair system.

[0015] Further, after the persistent file system completes the write file request, writing the target data information corresponding to the write file request into the key-value pair system includes: determining the second block number based on the second file name, the second start byte, and the length of the target data information in the write file request; determining whether the client stores the second block content corresponding to the second block number based on the second file name and the second block number; if the client stores the second block content corresponding to the second block number, then the target data information is used to overwrite the second block content to obtain the processed second block content, and the processed second block content is marked as the target dirty block; if the target data information has completely overwritten the second block content, then the client sends a message to the key-value pair system. The key-value pair system initiates a write request for the target dirty block to write the target dirty block into the key-value pair system. If the client does not store the second block content corresponding to the second block number, it determines whether the length of the target data information is equal to a preset length. If the length of the target data information is equal to the preset length, the target data information is written into the key-value pair system. If the length of the target data information is not equal to the preset length, the second block content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is overwritten on the client to obtain the processed second block content, and the processed second block content is marked as the target dirty block.

[0016] Further, if the data processing request type is a directory processing request, the data processing of the file data according to the request type includes: if the directory processing request is detected as a request to add a first directory node, then the operation content of the request to add a first directory node is added to the key-value pair corresponding to the first directory; if the directory processing request is detected as a request to delete a second directory node, then the operation content of the request to delete a second directory node is added to the key-value pair corresponding to the second directory; if the directory processing request is detected as a request to query a third directory, then the operation log information corresponding to the third directory is obtained from the key-value pair system based on the name of the parent node of the third directory; the operation log information is filtered to obtain the filtered third directory, and the filtered third directory is fed back to the application through the client.

[0017] According to one aspect of the embodiments of this application, a file processing apparatus is provided, comprising: an acquisition unit for acquiring a data processing request initiated by an application for file data; a determination unit for determining the request type of the data processing request, wherein the request type is one of the following: a read file request, a write file request, and a directory processing request; and a processing unit for performing data processing on the file data according to the request type.

[0018] Furthermore, if the data processing request is a file read request, the processing unit includes: a determining module, configured to determine a first block number based on the first file name, the first starting byte, and the length of the data to be read in the file read request; and an obtaining module, configured to obtain the first block content based on the first file name and the first block number, and to feed back the first block content to the application.

[0019] Further, the acquisition module includes: a first judgment submodule, used to determine whether the client stores the first block content based on the first file name and the first block number; a first acquisition submodule, used to obtain the first block content from the client if the client stores the first block content, and feed the first block content back to the application; a first forwarding submodule, used to forward the file read request to the key-value pair system through the client if the client does not store the first block content, and determine whether the key-value pair system stores the first block content based on the first file name and the first block number; a second acquisition submodule, used to cache the first block content stored in the key-value pair system to the client if the key-value pair system stores the first block content, and feed the first block content back to the application through the client; a second forwarding submodule, used to forward the file read request to the persistent file system through the client if the key-value pair system does not store the first block content, obtain the first block content from the persistent file system, and feed the first block content back to the application through the client.

[0020] Furthermore, the device further includes a storage unit, configured to store the first block of content in the form of key-value pairs in the key-value pair system via the client after the first block of content is fed back to the application via the client.

[0021] Furthermore, if the data processing request is a write file request, the processing unit includes: a third forwarding submodule, used to forward the write file request to a persistent file system via a client, and to complete the write file request in the persistent file system; and a writing module, used to write the target data information corresponding to the write file request into a key-value pair system after the persistent file system completes the write file request.

[0022] Further, the writing module includes: a determining submodule, used to determine a second block number based on the second file name, the second starting byte, and the length of the target data information in the write file request; a second judging submodule, used to judge whether the client has stored the second block content corresponding to the second block number based on the second file name and the second block number; and a first writing submodule, used to, if the client has stored the second block content corresponding to the second block number, execute the process of overwriting the second block content with the target data information to obtain the processed second block content, and mark the processed second block content as a target dirty block; if the target data information has completely overwritten the second block content, then initiate the writing process to the key-value pair system through the client. A write request for a target dirty block is made to write the target dirty block into the key-value pair system; a second write submodule is used to determine whether the length of the target data information is equal to a preset length if the client does not store the second block content corresponding to the second block number; if the length of the target data information is equal to the preset length, the target data information is written into the key-value pair system; if the length of the target data information is not equal to the preset length, the second block content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is overwritten on the client to obtain the processed second block content, and the processed second block content is marked as the target dirty block.

[0023] Further, if the data processing request is a directory processing request, the processing unit includes: a first detection module, configured to add the operation content of the add first directory node request to the key-value pair corresponding to the first directory if the directory processing request is detected as an add first directory node request; a second detection module, configured to add the operation content of the delete second directory node request to the key-value pair corresponding to the second directory if the directory processing request is detected as a delete second directory node request; a third detection module, configured to obtain the operation log information corresponding to the third directory from the key-value pair system based on the name of the parent node of the third directory if the directory processing request is detected as a query third directory request; and a filtering module, configured to filter the operation log information to obtain the filtered third directory, and feed the filtered third directory back to the application through the client.

[0024] In this embodiment, a client is employed, in which an application is deployed. The application initiates file data processing requests to the key-value pair system and / or persistent system through the client to provide services externally. The key-value pair system serves as a cache database between the client and the persistent file system. The persistent file system is used for persistent storage of file data. The application interacts with the client for file data through a portable operating system interface, and the client interacts with the key-value pair system through an interface adapted to the key-value pair system. The client also interacts with the persistent file system through the portable operating system interface. This solves the technical problem in related technologies where modifying the application's source code is required to use the key-value pair system, leading to a high error rate during data processing. The file processing system provided by this invention improves the application's data processing efficiency and reduces the error rate during data processing without modifying the application's source code. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 This is a schematic diagram of a document processing system provided according to Embodiment 1 of this application;

[0027] Figure 2 This is a schematic diagram of a computer terminal provided according to Embodiment 2 of this application;

[0028] Figure 3 This is a flowchart of a document processing method according to Embodiment 2 of this application;

[0029] Figure 4 This is a schematic diagram of a document processing apparatus according to Embodiment 3 of this application;

[0030] Figure 5 This is a schematic diagram of a computer terminal provided according to Embodiment 4 of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0034] Key-value store: A storage system that stores a fixed-length or variable-length block of data using a key as an index.

[0035] Persistent file system: Under File System (UFS), an accelerated file system located below the data caching layer;

[0036] Dirty blocks: Data blocks that store dirty data are called dirty blocks. Compared with the original data, data that has been modified and is different from the original data is called dirty data.

[0037] Example 1

[0038] According to embodiments of this application, a file processing system is provided, such as... Figure 1 The diagram illustrates a file processing system. This system includes: a client, where an application is deployed; the application initiates file data processing requests to a key-value pair system and / or a persistent file system through the client to provide services; a key-value pair system, which serves as a cache database between the client and the persistent file system; and a persistent file system, used for persistently storing file data. The application interacts with the client via a portable operating system interface, the client interacts with the key-value pair system via an interface adapted to the key-value pair system, and the client interacts with the persistent file system via the portable operating system interface.

[0039] Specifically, such as Figure 1As shown, the file processing system includes a client, a key-value pair system, and a persistent file system. The application is deployed on the client, which interacts with the application through the Portable Operating System Interface (POSIX API). The client translates the application's file access logic (i.e., file data processing logic) into get, put, and append operations on the key-value pair system. In other words, the client interacts with the key-value pair system through an interface adapted to the key-value pair system. The client uses the POSIX API interface to interact with the persistent file system. The key-value pair system provides get(), put(), and append() interfaces. The get() interface corresponds to read operations; the put() interface corresponds to write operations; and the append() interface corresponds to append operations. The key-value pair system acts as a cache database between the application and the persistent file system, improving the data processing efficiency of the application's file I / O ports. The persistent file system is mounted on the client as a data source, providing persistent data storage. With the file processing system described above, the application's source code can leverage the acceleration effect of the key-value pair system cache without modification. Furthermore, the key-value pair system does not have a central master node, which can effectively avoid single points of failure. It can also effectively improve the scalability of the file processing system and reduce the error rate of data processing.

[0040] In order to quickly query file data in the key-value pair system, in the file processing system provided in Embodiment 1 of the present invention, the files in the key-value pair system are stored in the form of multiple blocks of preset length. The file name and the block number of the block content are used as keys, and the block content is used as values ​​to form a first key-value pair, and the first key-value pair is stored in the key-value pair system.

[0041] Specifically, the file is stored in a key-value pair system with blocks of a preset length, meaning that the data is processed as a whole block. The length of the block can be set according to the actual situation, such as 2M or 4M. The first key-value pair is constructed by using the string consisting of the file name (i.e., the total path of the file) and the block number as the key and the block content as the value, and stored in the key-value pair system. For example, the 10th block of a file named a / b / c / x.txt has the key "a / b / c / x.txt"

[10] . By storing the file in the above manner, only the file name and the corresponding block number are needed to accurately query the data when reading it, which improves the efficiency of data query.

[0042] In practical use, file attribute information is also crucial. Therefore, in the file processing system provided in Embodiment 1 of this invention, the key-value pair system is also used to store file attribute information and symbolic links. The file name and attribute instructions are used as keys, and the file attribute information and / or the file symbolic links are used as values ​​to form a second key-value pair, which is then stored in the key-value pair system.

[0043] Specifically, file attribute information is stored in a key-value pair system as follows: the file name and attribute directive (e.g., " / a / b / c / x.txt[attri]") are used as keys, and file attribute information and / or symbolic links are used as values. Here, R can represent the attributes of a regular file, and L can represent a symbolic link. For example, the attribute information of file 1 is R:uid (file owner's ID):gid (user's ID):mode (file permission information):length (file length); the symbolic link format is L:uid:gid:mode:target (the target file the symbolic link points to). With this storage format, when querying file attributes, only the file name and [attri] are needed to accurately retrieve the file's attribute information, improving data processing efficiency.

[0044] The directory information of the file can be used to accurately understand and grasp all the data information currently stored. Therefore, in the file processing system provided in Embodiment 1 of the present invention, the key-value pair system is also used to store the directory tree of the file. The parent node name of the directory tree is used as the key, and the operation log information of the parent node is used as the value to form a component as a third key-value pair, and the third key-value pair is stored in the key-value pair system.

[0045] Specifically, directories are also stored using key-value pairs, with the name of the parent node as the key and the operation log information of the parent node as the value. Since the key-value pair system does not use a central master node and there are concurrent CRUD operations on a single parent node, operations on the directory are implemented by appending operation log information.

[0046] In summary, the file processing system provided by this invention solves the technical problem in related technologies where modifying and adapting the application's source code to use a key-value pair system leads to a high error rate during data processing. The file processing system described above can improve the application's data processing efficiency and reduce the error rate by utilizing a key-value pair system without modifying the application's source code.

[0047] Example 2

[0048] According to an embodiment of this application, a method for processing a document is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0049] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 2 A hardware block diagram of a computer terminal (or mobile device) for implementing a file processing method is shown. Figure 2 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0050] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or wholly or partially integrated into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0051] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the file processing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned file processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0053] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0054] Under the aforementioned operating environment, this application provides the following: Figure 3 The document shows the method for processing files. Figure 3 This is a flowchart of a document processing method according to Embodiment 2 of this application. The document processing method is applied to any of the document processing systems in Embodiment 1.

[0055] Step S301: Obtain the data processing request initiated by the application for file data.

[0056] Step S302: Determine the request type of the data processing request, wherein the request type is one of the following: read file request, write file request, and directory processing request.

[0057] Specifically, the application initiates a data processing request and determines the request type. Request types include: reading files, writing files, and directory processing. Request types may also include querying file attribute information.

[0058] Step S303: Process the file data according to the request type.

[0059] Specifically, data processing is performed based on the request type.

[0060] In summary, the application's source code does not need to be modified. As long as a data processing request is initiated normally, the aforementioned file processing system can accurately process the data according to the request type, thereby improving data processing efficiency.

[0061] When the data processing request type is a file read request, the file processing method provided in Embodiment 2 of the present invention uses the following steps to process the data: determine the first block number based on the first file name, the first starting byte and the length of the data to be read in the file read request; obtain the first block content based on the first file name and the first block number, and feed the first block content back to the application.

[0062] Specifically, since files in a key-value pair system are stored in block content format, with the file name and block number as the keys, the file read request includes the file name to be read (i.e., the first file name mentioned above), the first start byte, and the length of the data to be read. The first block number is determined using the first start byte and the length of the data to be read. It should be noted that there can be one or more first block numbers. Then, based on the first file name and the first block number, the first block of content required by the application is obtained. Through these steps, the target data to be read can be obtained quickly and accurately, reducing the error rate in data processing.

[0063] To accurately and quickly obtain the first block content corresponding to the first block number, the file processing method provided in Embodiment 2 of the present invention further limits the process of obtaining the first block content based on the first file name and the first block number, and feeding the first block content back to the application. The method uses the following steps to obtain the corresponding first block content: Based on the first file name and the first block number, determine whether the client stores the first block content; if the client stores the first block content, obtain the first block content from the client and feed it back to the application; if the client does not store the first block content, forward the file read request to the key-value pair system through the client, and determine whether the key-value pair system stores the first block content based on the first file name and the first block number; if the key-value pair system stores the first block content, cache the first block content stored in the key-value pair system to the client, and feed the first block content back to the application through the client; if the key-value pair system does not store the first block content, forward the file read request to the persistent file system through the client, obtain the first block content from the persistent file system, and feed the first block content back to the application through the client.

[0064] Specifically, firstly, since the client stores the most recently accessed and processed file data, it determines whether the client has the first block of content based on the first file name and the first block number. If the client has the first block of content, it sends it back to the application. If the client does not have the first block of content, it forwards the file read request to the key-value pair system to check if the first block of content is stored there. If the key-value pair system has the first block of content, it caches it on the client and sends it back to the application. If the key-value pair system does not have the first block of content, it forwards the file read request to the persistent file system, reads the first block of content from the persistent file system, and sends it back to the application.

[0065] By following the steps above, the first piece of content is read from the client when it is already stored on the client; the first piece of content is read from the key-value pair system when it is not stored on the client but is stored on the key-value pair system; and the first piece of content is read from the persistent file system when neither the client nor the key-value pair system stores it. This improves the efficiency and accuracy of reading files.

[0066] In the case that the first block of content is not stored in the key-value pair system, in order to quickly read the first block of content from the key-value pair system next time, the file processing method provided in Embodiment 2 of the present invention also needs to perform the following steps: the first block of content is stored in the key-value pair system in the form of key-value pairs by the client.

[0067] Specifically, after caching the first block of content in the persistent file system to the client, the client sends a put request to the key-value pair system to store the first block of content in the key-value pair system. This operation is so that when the first block of content is read again, it does not need to be read from the persistent file processing system, thus improving the efficiency of the application in reading data.

[0068] When the data processing request type is a write file request, the file processing method provided in Embodiment 2 of the present invention adopts the following steps for data processing: the write file request is forwarded to the persistent file system through the client, and the write file request is completed in the persistent file system; after the write file request is completed in the persistent file system, the target data information corresponding to the write file request is written to the key-value pair system.

[0069] Specifically, when an application initiates a file write request, the client uses write penetration to directly forward the write request to the persistent file system, where the write request is completed. Then, the target data information corresponding to the write request is written to the key-value pair system. This operation is performed so that if the target data information needs to be read later, it can be directly read from the key-value pair system, saving data processing time.

[0070] To ensure that target data information can be directly read from the key-value pair system when needed later, the file processing method provided in Embodiment 2 of this invention employs the following steps to write the target data information corresponding to the write file request into the key-value pair system: determining the second block number based on the second file name, the second starting byte, and the length of the target data information in the write file request; determining whether the client stores the second block content corresponding to the second block number based on the second file name and the second block number; if the client stores the second block content corresponding to the second block number, then the target data information is used to overwrite the second block content to obtain the processed second block content, and the processed second block content is marked as the target dirty block; if the target data information is not stored, the target data information is overwritten to obtain the processed second block content. If the target data information has completely covered the second block content, the client initiates a write request for the target dirty block to the key-value pair system to write the target dirty block into the key-value pair system. If the client does not store the second block content corresponding to the second block number, it determines whether the length of the target data information is equal to the preset length. If the length of the target data information is equal to the preset length, the target data information is written into the key-value pair system. If the length of the target data information is not equal to the preset length, the second block content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is used to overwrite the second block content on the client to obtain the processed second block content, which is then marked as the target dirty block.

[0071] Specifically, the second block number is determined based on the second file name, the second start byte, and the length of the target data information in the write file request. The system then checks whether the client has stored the second block content corresponding to the second block number. If the client has stored the second block content, the target data information overwrites the client's second block content, resulting in a processed second block content, which is then marked as a dirty block. The system further checks whether the target data information completely overwrites the second block content. If it does, the client initiates a put request to the key-value pair system, writing the dirty block to the system. However, if the target data information does not completely overwrite the second block content, it is not written to the key-value pair system. If the client has not stored the second block content, the system checks whether the target data information equals the preset length (which is the length of the second block content). If the target data information equals the preset length, it is directly written to the key-value pair system. If the target data is not equal to the preset length, the second block of content in the key-value pair system is cached on the client, and the target data is used to overwrite the second block of content in the client, resulting in a processed second block of content. This processed second block of content is then marked as a dirty target block. These steps are to ensure that if the target data needs to be read later, it can be directly read from the key-value pair system, thereby improving the application's I / O throughput and reducing I / O latency.

[0072] When the data processing request type is a directory processing request, the file processing method provided in Embodiment 2 of the present invention employs the following steps for data processing: If the detected directory processing request is a request to add a first directory node, the operation content of the request to add a first directory node is added to the key-value pair corresponding to the first directory; if the detected directory processing request is a request to delete a second directory node, the operation content of the request to delete a second directory node is added to the key-value pair corresponding to the second directory; if the detected directory processing request is a request to query a third directory, the operation log information corresponding to the third directory is obtained from the key-value pair system based on the name of the parent node of the third directory; the operation log information is filtered to obtain the filtered third directory, and the filtered third directory is fed back to the application through the client.

[0073] Specifically, when adding a node to a directory (e.g., creating a file or using the `mkdir` command), the client sends an `append` operation to the key-value pair system. Based on the name of the directory's parent node, operation log information is added to the key-value pair corresponding to that directory. The added content can be: "<marker character>child node name", where the marker character is F, L, or D. F, L, and D correspond to the type of the newly added child node: ordinary file, symbolic link, and directory, respectively.

[0074] When a node is deleted from a directory (e.g., deleting a file or using the rmdir command), the client sends an append operation to the key-value pair system. Based on the name of the parent node of the directory, operation log information is added to the key-value pair corresponding to the directory. The added content can be "X child node name", where X represents the deletion operation.

[0075] When reading a directory (e.g., using the `ls` operation), the client sends a `get` operation to the key-value pair system. Based on the name of the directory's parent node, the system retrieves all operation logs for that directory. Then, it scans the logs for marker characters (i.e., the 'X' mentioned above) and calculates the remaining valid child nodes. Finally, the filtered directory is sent back to the application.

[0076] It should be noted that, in order to clean up junk in the logs, specifically log entries marked with an "X", the key-value pair system needs to perform a cleanup task periodically. This involves organizing key-value pairs with "X" log entries exceeding a threshold and eliminating invalid log entries marked with "X". This improves the efficiency of subsequent directory queries.

[0077] In the file processing method provided in Embodiment 2 of the present invention, the data processing request initiated by the application for file data is obtained; the request type of the data processing request is determined, wherein the request type is one of the following: read file request, write file request, and directory processing request; and the file data is processed according to the request type. This solves the technical problem in related technologies that the source code of the application needs to be modified and adapted to use a key-value pair system, resulting in a relatively high error rate during data processing. The file processing system provided by the present invention can improve the data processing efficiency of the application by utilizing a key-value pair system without modifying the application's source code, and also achieves the effect of reducing the error rate during data processing.

[0078] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0080] Example 3

[0081] According to embodiments of this application, a file processing apparatus for implementing the above-described file processing method is also provided, such as... Figure 4 As shown, the device includes: an acquisition unit 401, a determination unit 402, and a processing unit 403.

[0082] The acquisition unit 401 is used to acquire data processing requests initiated by the application for file data.

[0083] The determining unit 402 is used to determine the request type of the data processing request, wherein the request type is one of the following: read file request, write file request, and directory processing request.

[0084] Processing unit 403 is used to process file data according to the request type.

[0085] In summary, in the file processing apparatus provided in Embodiment 3 of the present invention, the acquisition unit 401 acquires the data processing request initiated by the application for file data; the determination unit 402 determines the request type of the data processing request, wherein the request type is one of the following: a read file request, a write file request, and a directory processing request; and the processing unit 403 processes the file data according to the request type. This solves the technical problem in related technologies where the source code of the application needs to be modified and adapted to use a key-value pair system, resulting in a relatively high error rate during data processing. Through the file processing system provided by the present invention, the data processing efficiency of the application can be improved by utilizing a key-value pair system without modifying the application's source code, and the error rate during data processing is reduced.

[0086] Optionally, in the file processing apparatus provided in Embodiment 3 of the present invention, if the type of data processing request is a file read request, the processing unit 403 includes: a determining module, used to determine a first block number based on the first file name, the first starting byte and the length of the data to be read in the file read request; and an obtaining module, used to obtain the first block content based on the first file name and the first block number, and to feed back the first block content to the application.

[0087] Optionally, in the file processing apparatus provided in Embodiment 3 of the present invention, the acquisition module includes: a first judgment submodule, used to determine whether the client stores the first block of content based on the first file name and the first block number; a first acquisition submodule, used to obtain the first block of content from the client if the client stores the first block of content, and to feed the first block of content back to the application; a first forwarding submodule, used to forward the file read request to the key-value pair system through the client if the client does not store the first block of content, and to determine whether the key-value pair system stores the first block of content based on the first file name and the first block number; a second acquisition submodule, used to cache the first block of content stored in the key-value pair system to the client if the key-value pair system stores the first block of content, and to feed the first block of content back to the application through the client; and a second forwarding submodule, used to forward the file read request to the persistent file system through the client if the key-value pair system does not store the first block of content, to obtain the first block of content from the persistent file system, and to feed the first block of content back to the application through the client.

[0088] Optionally, in the file processing apparatus provided in Embodiment 3 of the present invention, the apparatus further includes: a storage unit, used to store the first block of content in the form of a key-value pair system by the client after the client feeds back the first block of content to the application.

[0089] Optionally, in the file processing apparatus provided in Embodiment 3 of the present invention, if the type of data processing request is a write file request, the processing unit 403 includes: a third forwarding submodule, used to forward the write file request to the persistent file system through the client, and complete the write file request in the persistent file system; and a writing module, used to write the target data information corresponding to the write file request into the key-value pair system after the persistent file system completes the write file request.

[0090] Optionally, in the file processing apparatus provided in Embodiment 3 of the present invention, the writing module includes: a determining submodule, configured to determine a second block number based on the second file name, the second starting byte, and the length of the target data information in the file writing request; a second judging submodule, configured to judge whether the client has stored the second block content corresponding to the second block number based on the second file name and the second block number; and a first writing submodule, configured to, if the client has stored the second block content corresponding to the second block number, execute the process of overwriting the second block content with the target data information to obtain the processed second block content, and mark the processed second block content as the target dirty block; if the target data information has completely overwritten the second block content, then... The client initiates a write request to the key-value pair system to write the target dirty block into the key-value pair system. The second write submodule is used to determine whether the length of the target data information is equal to the preset length if the client does not store the second block content corresponding to the second block number. If the length of the target data information is equal to the preset length, the target data information is written to the key-value pair system. If the length of the target data information is not equal to the preset length, the second block content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is used to overwrite the second block content on the client to obtain the processed second block content. The processed second block content is then marked as the target dirty block.

[0091] Optionally, in the file processing apparatus provided in Embodiment 3 of the present invention, if the type of data processing request is a directory processing request, the processing unit 403 includes: a first detection module, configured to add the operation content of the first directory node request to the key-value pair corresponding to the first directory if the detected directory processing request is a request to add a first directory node; a second detection module, configured to add the operation content of the second directory node request to the key-value pair corresponding to the second directory if the detected directory processing request is a request to delete a second directory node; a third detection module, configured to obtain the operation log information corresponding to the third directory from the key-value pair system based on the name of the parent node of the third directory if the detected directory processing request is a request to query a third directory; and a filtering module, configured to filter the operation log information to obtain the filtered third directory, and feed the filtered third directory back to the application through the client.

[0092] It should be noted that the acquisition unit 401, the determination unit 402, and the processing unit 403 mentioned above correspond to steps S301 to S303 in Embodiment 2. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0093] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0094] Example 4

[0095] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0096] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0097] In this embodiment, the computer terminal described above can execute the program code for the following steps in the file processing method: obtaining a data processing request initiated by an application for file data; determining the request type of the data processing request, wherein the request type is one of the following: a read file request, a write file request, and a directory processing request; and processing the file data according to the request type.

[0098] The aforementioned computer terminal can also execute program code for the following steps in the file processing method: If the type of the data processing request is a file read request, the data processing of the file data according to the request type includes: determining the first block number based on the first file name, the first starting byte, and the length of the data to be read in the file read request; obtaining the first block content based on the first file name and the first block number, and feeding back the first block content to the application program.

[0099] The aforementioned computer terminal can also execute program code for the following steps in the file processing method: obtaining the first block of content based on the first file name and the first block number, and feeding the first block of content back to the application, including: determining whether the client has stored the first block of content based on the first file name and the first block number; if the client has stored the first block of content, obtaining the first block of content from the client and feeding the first block of content back to the application; if the client has not stored the first block of content, forwarding the file read request to the key-value pair system through the client, and determining whether the key-value pair system has stored the first block of content based on the first file name and the first block number; if the key-value pair system has stored the first block of content, caching the first block of content stored in the key-value pair system to the client, and feeding the first block of content back to the application through the client; if the key-value pair system has not stored the first block of content, forwarding the file read request to the persistent file system through the client, obtaining the first block of content from the persistent file system, and feeding the first block of content back to the application through the client.

[0100] The aforementioned computer terminal can also execute program code for the following steps in the file processing method: after the first piece of content is fed back to the application through the client, the method further includes: storing the first piece of content in the form of key-value pairs to the key-value pair system through the client.

[0101] The aforementioned computer terminal can also execute program code for the following steps in the file processing method: if the type of the data processing request is a write file request, the data processing of the file data according to the request type includes: forwarding the write file request to the persistent file system through the client, completing the write file request in the persistent file system; after the write file request is completed in the persistent file system, writing the target data information corresponding to the write file request into the key-value pair system.

[0102] The aforementioned computer terminal can also execute program code for the following steps in the file processing method: After the persistent file system completes the write file request, writing the target data information corresponding to the write file request into the key-value pair system includes: determining the second block number based on the second file name, the second starting byte, and the length of the target data information in the write file request; determining whether the client stores the second block content corresponding to the second block number based on the second file name and the second block number; if the client stores the second block content corresponding to the second block number, then executing the process of overwriting the second block content with the target data information to obtain the processed second block content, and marking the processed second block content as the target dirty block; if the target data information... If the second block of content has been completely overwritten, the client initiates a write request for the target dirty block to the key-value pair system to write the target dirty block into the key-value pair system. If the client does not store the second block of content corresponding to the second block number, it determines whether the length of the target data information is equal to the preset length. If the length of the target data information is equal to the preset length, the target data information is written to the key-value pair system. If the length of the target data information is not equal to the preset length, the second block of content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is used to overwrite the second block of content on the client to obtain the processed second block of content, which is then marked as the target dirty block.

[0103] The aforementioned computer terminal can also execute program code for the following steps in the file processing method: If the data processing request type is a directory processing request, the data processing of the file data according to the request type includes: if the detected directory processing request is a request to add a first directory node, then the operation content of the request to add a first directory node is added to the key-value pair corresponding to the first directory; if the detected directory processing request is a request to delete a second directory node, then the operation content of the request to delete a second directory node is added to the key-value pair corresponding to the second directory; if the detected directory processing request is a request to query a third directory, then the operation log information corresponding to the third directory is obtained from the key-value pair system based on the name of the parent node of the third directory; the operation log information is filtered to obtain the filtered third directory, and the filtered third directory is fed back to the application through the client.

[0104] Optionally, Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 5 As shown, the computer terminal 10 may include: one or more ( Figure 5 (Only one is shown in the image) processor and memory.

[0105] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the file processing method and apparatus in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the aforementioned file processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0106] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtain data processing requests initiated by the application for file data; determine the request type of the data processing request, wherein the request type is one of the following: read file request, write file request, and directory processing request; and process the file data according to the request type.

[0107] Optionally, the processor may also execute program code with the following steps: if the data processing request type is a file read request, the data processing of the file data according to the request type includes: determining the first block number based on the first file name, the first starting byte and the length of the data to be read in the file read request; obtaining the first block content based on the first file name and the first block number, and feeding back the first block content to the application.

[0108] Optionally, the processor may also execute program code with the following steps: obtaining the first block of content based on the first file name and the first block number, and feeding the first block of content back to the application, including: determining whether the client has stored the first block of content based on the first file name and the first block number; if the client has stored the first block of content, obtaining the first block of content from the client and feeding the first block of content back to the application; if the client has not stored the first block of content, forwarding the file read request to the key-value pair system through the client, and determining whether the key-value pair system has stored the first block of content based on the first file name and the first block number; if the key-value pair system has stored the first block of content, caching the first block of content stored in the key-value pair system to the client, and feeding the first block of content back to the application through the client; if the key-value pair system has not stored the first block of content, forwarding the file read request to the persistent file system through the client, obtaining the first block of content from the persistent file system, and feeding the first block of content back to the application through the client.

[0109] Optionally, the processor may also execute program code that performs the following steps: after the first piece of content is fed back to the application via the client, the method further includes: storing the first piece of content in the form of key-value pairs to the key-value pair system via the client.

[0110] Optionally, the processor may also execute program code with the following steps: if the data processing request type is a write file request, the data processing of the file data according to the request type includes: forwarding the write file request to the persistent file system through the client, completing the write file request in the persistent file system; after the write file request is completed in the persistent file system, writing the target data information corresponding to the write file request into the key-value pair system.

[0111] Optionally, the processor may also execute program code with the following steps: After the persistent file system completes the write file request, writing the target data information corresponding to the write file request into the key-value pair system includes: determining the second block number based on the second file name, the second start byte, and the length of the target data information in the write file request; determining whether the client stores the second block content corresponding to the second block number based on the second file name and the second block number; if the client stores the second block content corresponding to the second block number, then overwriting the second block content with the target data information to obtain the processed second block content, and marking the processed second block content as the target dirty block; if the target data information is complete... If the second block of content is to be overwritten, the client initiates a write request for the target dirty block to the key-value pair system to write the target dirty block into the key-value pair system. If the client does not store the second block of content corresponding to the second block number, it is determined whether the length of the target data information is equal to the preset length. If the length of the target data information is equal to the preset length, the target data information is written to the key-value pair system. If the length of the target data information is not equal to the preset length, the second block of content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is used to overwrite the second block of content on the client to obtain the processed second block of content, which is then marked as the target dirty block.

[0112] Optionally, the processor may also execute program code with the following steps: If the data processing request type is a directory processing request, the data processing of the file data according to the request type includes: if the detected directory processing request is a request to add a first directory node, then the operation content of the request to add a first directory node is added to the key-value pair corresponding to the first directory; if the detected directory processing request is a request to delete a second directory node, then the operation content of the request to delete a second directory node is added to the key-value pair corresponding to the second directory; if the detected directory processing request is a request to query a third directory, then the operation log information corresponding to the third directory is obtained from the key-value pair system according to the name of the parent node of the third directory; the operation log information is filtered to obtain the filtered third directory, and the filtered third directory is fed back to the application through the client.

[0113] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0114] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0115] Example 5

[0116] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the file processing method provided in Embodiment 1.

[0117] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0118] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a data processing request initiated by an application for file data; determining the request type of the data processing request, wherein the request type is one of the following: a read file request, a write file request, and a directory processing request; and performing data processing on the file data according to the request type.

[0119] The aforementioned storage medium is also configured to store program code for performing the following steps: if the type of the data processing request is a file read request, the data processing of the file data according to the request type includes: determining the first block number based on the first file name, the first starting byte, and the length of the data to be read in the file read request; obtaining the first block content based on the first file name and the first block number, and feeding back the first block content to the application.

[0120] The aforementioned storage medium is also configured to store program code for performing the following steps: obtaining a first block of content based on a first file name and a first block number, and feeding the first block of content back to the application, including: determining whether the client has stored the first block of content based on the first file name and the first block number; if the client has stored the first block of content, obtaining the first block of content from the client and feeding the first block of content back to the application; if the client has not stored the first block of content, forwarding the file read request to the key-value pair system through the client, and determining whether the key-value pair system has stored the first block of content based on the first file name and the first block number; if the key-value pair system has stored the first block of content, caching the first block of content stored in the key-value pair system to the client, and feeding the first block of content back to the application through the client; if the key-value pair system has not stored the first block of content, forwarding the file read request to the persistent file system through the client, obtaining the first block of content from the persistent file system, and feeding the first block of content back to the application through the client.

[0121] The aforementioned storage medium is also configured to store program code for performing the following steps: after the first block of content is fed back to the application via the client, the method further includes: storing the first block of content as a key-value pair to the key-value pair system via the client.

[0122] The aforementioned storage medium is also configured to store program code for performing the following steps: if the type of the data processing request is a write file request, the data processing of the file data according to the request type includes: forwarding the write file request to the persistent file system through the client, completing the write file request in the persistent file system; after the write file request is completed in the persistent file system, writing the target data information corresponding to the write file request into the key-value pair system.

[0123] The aforementioned storage medium is also configured to store program code for performing the following steps: After the persistent file system completes the write file request, writing the target data information corresponding to the write file request into the key-value pair system includes: determining the second block number based on the second file name, the second start byte, and the length of the target data information in the write file request; determining whether the client has stored the second block content corresponding to the second block number based on the second file name and the second block number; if the client has stored the second block content corresponding to the second block number, then executing the overwriting of the second block content with the target data information to obtain the processed second block content, and marking the processed second block content as the target dirty block; if the target data information has been completed... If the second block of content is completely overwritten, the client initiates a write request for the target dirty block to the key-value pair system to write the target dirty block into the key-value pair system. If the client does not store the second block of content corresponding to the second block number, it determines whether the length of the target data information is equal to the preset length. If the length of the target data information is equal to the preset length, the target data information is written to the key-value pair system. If the length of the target data information is not equal to the preset length, the second block of content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is overwritten on the client to obtain the processed second block of content, which is then marked as the target dirty block.

[0124] The aforementioned storage medium is also configured to store program code for performing the following steps: If the type of the data processing request is a directory processing request, the data processing of the file data according to the request type includes: if the detected directory processing request is a request to add a first directory node, then the operation content of the request to add a first directory node is added to the key-value pair corresponding to the first directory; if the detected directory processing request is a request to delete a second directory node, then the operation content of the request to delete a second directory node is added to the key-value pair corresponding to the second directory; if the detected directory processing request is a request to query a third directory, then the operation log information corresponding to the third directory is obtained from the key-value pair system based on the name of the parent node of the third directory; the operation log information is filtered to obtain the filtered third directory, and the filtered third directory is fed back to the application through the client.

[0125] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0126] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0131] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A file processing system, characterized in that, include: A client, wherein an application is deployed in the client, and the application initiates requests for file data processing to the key-value pair system and / or persistent system through the client in order to provide services to the outside world; The key-value pair system serves as a cache database between the client and the persistent file system. The persistent file system is used for persistently storing file data. The application interacts with the client via a portable operating system interface, the client interacts with the key-value pair system via an interface adapted to the key-value pair system, and the client interacts with the persistent file system via a portable operating system interface.

2. The file processing system according to claim 1, characterized in that, The files in the key-value pair system are stored in the form of multiple blocks of preset length. The file name and the block number of the block content are used as keys, and the block content is used as values ​​to form a first key-value pair, which is then stored in the key-value pair system.

3. The document processing system according to claim 1, characterized in that, The key-value pair system is also used to store file attribute information and symbolic links, wherein the file name and attribute instructions are used as keys, and the file attribute information and / or the file symbolic links are used as values ​​to form a second key-value pair, and the second key-value pair is stored in the key-value pair system.

4. The file processing system according to claim 1, characterized in that, The key-value pair system is also used to store the directory tree of the file, wherein a third key-value pair is constructed by using the name of the parent node of the directory tree as the key and the operation log information of the parent node as the value, and the third key-value pair is stored in the key-value pair system.

5. A method for processing files, characterized in that, The processing method is applied to the file processing system according to any one of claims 1-4, comprising: Get the data processing requests initiated by the application for file data; Determine the request type of the data processing request, wherein the request type is one of the following: read file request, write file request, and directory processing request; The file data is processed according to the request type.

6. The processing method according to claim 5, characterized in that, If the data processing request is a file read request, the data processing of the file data according to the request type includes: The first block number is determined based on the first file name, the first starting byte, and the length of the data to be read in the file read request; Based on the first file name and the first block number, the first block content is obtained and then fed back to the application.

7. The processing method according to claim 6, characterized in that, Based on the first file name and the first block number, obtaining the first block content and feeding the first block content back to the application includes: Based on the first file name and the first block number, determine whether the client has stored the first block content; If the client stores the first block of content, the application will retrieve the first block of content from the client and send it back to the application. If the client does not store the first block of content, the client forwards the file read request to the key-value pair system, and determines whether the key-value pair system stores the first block of content based on the first file name and the first block number. If the key-value pair system stores the first block of content, then the first block of content stored in the key-value pair system is cached in the client, and the first block of content is fed back to the application through the client; If the key-value pair system does not store the first block of content, the client forwards the file read request to the persistent file system, retrieves the first block of content from the persistent file system, and then sends the first block of content back to the application through the client.

8. The processing method according to claim 7, characterized in that, After the first piece of content is fed back to the application through the client, the method further includes: The client stores the first block of content in the form of key-value pairs into the key-value pair system.

9. The processing method according to claim 5, characterized in that, If the data processing request is a write file request, the data processing of the file data according to the request type includes: The client forwards the file write request to the persistent file system, where the file write request is completed. After the persistent file system completes the write file request, it writes the target data information corresponding to the write file request into the key-value pair system.

10. The processing method according to claim 9, characterized in that, After the persistent file system completes the write file request, writing the target data information corresponding to the write file request into the key-value pair system includes: The second block number is determined based on the second file name, the second start byte, and the length of the target data information in the write file request; Based on the second file name and the second block number, determine whether the client has stored the second block content corresponding to the second block number; If the client stores the second block content corresponding to the second block number, then the target data information is used to overwrite the second block content to obtain the processed second block content, and the processed second block content is marked as the target dirty block; if the target data information has completely overwritten the second block content, then the client initiates a write request for the target dirty block to the key-value pair system to write the target dirty block to the key-value pair system. If the client does not store the second block content corresponding to the second block number, it determines whether the length of the target data information is equal to a preset length. If the length of the target data information is equal to the preset length, the target data information is written into the key-value pair system. If the length of the target data information is not equal to the preset length, the second block content corresponding to the target data information is cached from the key-value pair system to the client, and the target data information is used to overwrite the second block content on the client to obtain the processed second block content. The processed second block content is then marked as the target dirty block.

11. The processing method according to claim 5, characterized in that, If the data processing request is a directory processing request, the data processing of the file data according to the request type includes: If the directory processing request is detected as a request to add a first directory node, then the operation content of the request to add a first directory node is added to the key-value pair corresponding to the first directory; If the directory processing request is detected as a request to delete a second directory node, then the operation content of the request to delete the second directory node is added to the key-value pair corresponding to the second directory. If the directory processing request is detected as a query request for a third directory, then the operation log information corresponding to the third directory is obtained from the key-value pair system based on the name of the parent node of the third directory; The operation log information is filtered to obtain a filtered third directory, and the filtered third directory is fed back to the application through the client.

12. A document processing apparatus, characterized in that, The processing apparatus is applied to the document processing system according to any one of claims 1-4, comprising: The acquisition unit is used to acquire data processing requests initiated by the application for file data; The determining unit is used to determine the request type of the data processing request, wherein the request type is one of the following: a file read request, a file write request, and a directory processing request; The processing unit is used to process the file data according to the request type.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the file processing method according to any one of claims 5 to 11.

Images