Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product. Comprising the steps of judging whether a transmission mode of an application layer for a to-be-written data write request is a direct read-write mode or not; if the mode is not the direct read-write mode, continuing to judge whether a client cache mechanism is started or not; if the client cache mechanism is started, continuing to judge whether the to-be-written data or metadata of a parent directory of the to-be-written data carry a cache acceleration mark or not; if the to-be-written data or the metadata of the parent directory of the to-be-written data carry a cache acceleration mark, obtaining the size of an idle memory block of a preset cache; when the size of the free memory block is greater than or equal to that of the to-be-written data and the to-be-written data is not overlapped with the data in the preset cache, storing the to-be-written data in a memory block pre-allocated by the preset cache; and traversing the memory block list of the preset cache, and storing the to-be-written data to the target storage. The method is used for achieving the effect of improving the data processing performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data processing method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of artificial intelligence and high-performance computing applications, there are higher performance requirements for the underlying distributed storage, especially in terms of latency, which usually needs to be controlled within the order of hundreds of microseconds to meet the requirements of real-time data interaction and large-scale parallel computing scenarios.

[0003] In the prior art, the Lustre distributed file system effectively avoids the additional overhead caused by frequent switching between the user state and the kernel state by deploying all services in the kernel state.

[0004] However, since each write operation needs to acquire a distributed lock to ensure metadata consistency, the application, verification and release process of the distributed lock involves communication and coordination between multiple nodes, which will produce an unavoidable latency superposition, resulting in insufficient overall performance. SUMMARY

[0005] The embodiments of the present application provide a data processing method, device, equipment, storage medium and program product to improve the performance of data processing.

[0006] In a first aspect, the embodiments of the present application provide a data processing method applied to a file system client, comprising: receiving a write request sent by an application layer for to-be-written data, judging whether the transmission mode of the write request is a direct read-write mode; if the transmission mode of the write request is not the direct read-write mode, continuing to judge whether a client cache mechanism is enabled for the file system client; if the client cache mechanism is enabled, continuing to judge whether a cache acceleration mark is carried in the metadata of the to-be-written data or the parent directory of the to-be-written data; if the cache acceleration mark is carried in the metadata of the to-be-written data or the parent directory of the to-be-written data, obtaining the size of a free memory block of a preset cache; judging whether the size of the free memory block is greater than or equal to the size of the to-be-written data and whether the to-be-written data and the data in the preset cache are not overlapped; if the size of the free memory block is greater than or equal to the size of the to-be-written data and the to-be-written data and the data in the preset cache are not overlapped, saving the to-be-written data into a memory block pre-allocated in the preset cache; and traversing a memory block list of the preset cache to save the to-be-written data into a target storage.

[0007] In one possible implementation, determining whether the metadata of the data to be written or the parent directory of the data to be written carries a cache acceleration tag includes: reading the metadata of the data to be written, wherein the metadata includes a cache acceleration status field; determining whether the cache acceleration status field matches the cache acceleration tag; if the cache acceleration status field of the data to be written matches the cache acceleration tag, determining that the metadata of the data to be written carries the cache acceleration tag; if the cache acceleration status field of the data to be written does not match the cache acceleration tag, obtaining the metadata of the direct parent directory to which the data to be written belongs, and determining whether the metadata of the direct parent directory carries the cache acceleration tag; if the metadata of the direct parent directory does not carry the cache acceleration tag, continuing to read and determine the metadata of the parent directory of the next level until traversing to the root directory; if the cache acceleration tag is matched in the metadata of the parent directory at any level, determining that the metadata of the parent directory of the data to be written carries the cache acceleration tag; if the cache acceleration tag is not matched in the metadata of the data to be written and the metadata of the parent directories of all levels, determining that neither the data to be written nor the metadata of the parent directory of the data to be written carries the cache acceleration tag.

[0008] In one possible implementation, after determining that neither the data to be written nor the metadata of the parent directory of the data to be written carry a cache acceleration mark, it also includes: sending an update request for the metadata of the data to be written to the metadata server, so that the metadata server writes the cache acceleration mark in the metadata of the data to be written and stores it persistently; receiving a mark write success response returned by the metadata server, and completing the addition of the cache acceleration mark in the metadata of the data to be written.

[0009] In one possible implementation, determining whether the size of a free memory block is greater than or equal to the size of the data to be written, and whether the data to be written has no overlap with the data in a preset cache includes: traversing a list of memory blocks in the preset cache, filtering out memory blocks in an idle state, and counting the actual available capacity of each idle memory block; comparing the sum of the actual available capacity of the filtered idle memory blocks with the size of the data to be written to determine whether there is a free memory block with a capacity that meets the requirement; if there is a free memory block with a capacity that meets the requirement, determining that the size of the free memory block is greater than or equal to the size of the data to be written; comparing the file identifier and data range information corresponding to the data for which target storage persistence has not been completed in the preset cache with the file identifier and data range information of the data to be written to determine whether there is an identical file identifier and an intersection in the data ranges; if there is an identical file identifier and an intersection in the data ranges, determining that the data to be written has overlap with the data in the preset cache.

[0010] In a possible implementation, the method further includes: receiving a read request sent by the application layer for the target data, determining whether a transmission mode of the read request is the direct read-write mode, if the transmission mode of the read request is not the direct read-write mode, checking whether the client-side cache mechanism is enabled for the file system client, if the client-side cache mechanism is enabled for the file system client, further checking whether the cache acceleration flag is carried in the metadata of the target data or a parent directory of the target data, if the cache acceleration flag is carried in the metadata of the target data or the parent directory of the target data, querying whether the target data exists in the preset cache, if the target data exists in the preset cache, reading the target data from the preset cache and returning the target data to the application layer, and if the target data does not exist in the preset cache, reading the target data from the target storage and saving the target data in a memory block pre-allocated in the preset cache.

[0011] In a second aspect, an embodiment of the present application provides a data processing apparatus applied to a file system client, comprising:

[0012] The transmission mode determination module is configured to receive a write request sent by the application layer for the to-be-written data, and determine whether a transmission mode of the write request is the direct read-write mode.

[0013] The cache mechanism enabling determination module is configured to, if the transmission mode of the write request is not the direct read-write mode, continue to determine whether the client-side cache mechanism is enabled for the file system client.

[0014] The cache acceleration flag determination module is configured to, if the client-side cache mechanism is enabled for the file system client, continue to determine whether the cache acceleration flag is carried in the metadata of the to-be-written data or a parent directory of the to-be-written data.

[0015] The idle memory size obtaining module is configured to, if the cache acceleration flag is carried in the metadata of the to-be-written data or the parent directory of the to-be-written data, obtain a size of an idle memory block of the preset cache.

[0016] The idle memory availability determination module is configured to determine whether the size of the idle memory block is greater than or equal to a size of the to-be-written data, and whether the to-be-written data and data in the preset cache are overlapped.

[0017] The data storage module is configured to, if the size of the idle memory block is greater than or equal to the size of the to-be-written data, and the to-be-written data and the data in the preset cache are not overlapped, save the to-be-written data in the memory block pre-allocated in the preset cache.

[0018] The persistent storage module is configured to traverse a memory block list of the preset cache, and save the to-be-written data to the target storage.

[0019] In a third aspect, an embodiment of the present application provides a file system client, comprising: a memory and a processor; the memory stores computer-executable instructions; and the processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0022] The data processing method, device, equipment, storage medium and program product provided by the embodiments of the present application can significantly improve the data write performance through multi-level judgment and precise cache control. First, invalid cache operations are avoided through the screening of the transmission mode and the cache mechanism; then, the waste of resources is reduced by precisely positioning the data to be accelerated through the hierarchical verification of the cache acceleration mark; at the same time, the cache safety is guaranteed by relying on the memory block management of the preset cache, combining the idle size and data overlap check, and greatly reducing the write latency by reducing the dependence on distributed locks through batch persistence, which greatly reduces the write latency and adapts to the demand for hundred-microsecond-level latency in high-performance computing scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] Figure 1 The architecture diagram of the file system client data processing flow provided by the embodiments of the present application is shown.

[0025] Figure 2 The flowchart of the data processing method provided by the embodiments of the present application is shown.

[0026] Figure 3 The structure diagram of the data processing device provided by the embodiments of the present application is shown.

[0027] Figure 4 The structure diagram of the file system client provided by the embodiments of the present application is shown.

[0028] Through the above drawings, the specific embodiments of the present application have been shown, and more detailed descriptions will be given in the following. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0030] To clearly understand the technical solutions of this application, we first provide a detailed introduction to existing solutions. With the rapid development of artificial intelligence and high-performance computing applications, higher performance requirements are being placed on underlying distributed storage, typically requiring latency within 100 microseconds. In traditional distributed storage, service processes typically run in user mode, requiring switching between user and kernel mode for network and disk I / O. This architecture fails to meet ultra-low latency requirements. The Lustre distributed file system was specifically designed for artificial intelligence and high-performance computing scenarios. All services in the Lustre distributed file system run in kernel mode, avoiding the overhead of switching between user and kernel mode and meeting the required ultra-low latency requirements. However, with the dramatic increase in the number of training parameters, the file size generated during training has also expanded. The speed of writing files directly impacts the speed of model training. Although files are written asynchronously after model training, the Lustre distributed file system also needs to obtain a distributed lock on the target storage when writing to the page cache. Due to the overhead of distributed locks, the Lustre distributed file system suffers from insufficient performance when writing files, resulting in single training runs taking minutes, making it difficult to meet the requirements of high-performance training.

[0031] To address the aforementioned technical issues, the inventors devised a hierarchical cache control mechanism based on cache acceleration tags on the file system client. This mechanism first determines the write request transmission mode, enabling client-side caching only for non-direct read / write scenarios. The mechanism then verifies the cache acceleration tags in the metadata of the data to be written or its parent directory to precisely locate the data to be accelerated. The mechanism then uses pre-allocated memory blocks to store the data, ensuring cache security by checking the size of the free memory blocks and the non-overlapping nature of the data. Finally, the cached data is written to the target storage in batches. This method reduces the frequency of reliance on distributed locks and maximizes write performance by combining a single memory copy mechanism. This approach not only retains the advantages of the Lustre distributed file system's kernel-mode services but also adapts to the ultra-low latency requirements of high-performance computing, effectively improving data write efficiency.

[0032] Figure 1 This is a schematic diagram of the architecture of the file system client data processing process provided in the embodiment of this application. Figure 1 As shown, the functions of each component are as follows:

[0033] The file system client, as an entry and control center of data processing, is responsible for receiving read requests or write requests initiated by the application layer, performing basic logic such as transmission mode judgment, client cache mechanism enabling verification, and coordinating data interaction and process control among components.

[0034] The cache acceleration module is the core carrier of the client cache acceleration strategy, used for temporarily storing data to be processed, and can accurately identify data that needs to be accelerated based on metadata markers, perform idle memory block arbitrage, data overlap verification, and other operations, and realize efficient caching and batch persistence scheduling of data.

[0035] The system page cache stores data in scenarios such as insufficient memory in the cache acceleration module or missing cache acceleration markers.

[0036] The metadata server stores file system metadata and provides metadata verification and update services for the file system client.

[0037] The management server maintains global information and provides basic support for metadata management and lock coordination in a distributed environment.

[0038] The target storage is the node where data is finally persisted and stored, and is the final landing point of the data processing flow, receiving persistent data from the cache acceleration module or the system cache.

[0039] Specifically, when the file system client receives a read or write request from the application layer, it first determines the data flow based on the transmission mode, cache mechanism enabling state, and other conditions. If the cache acceleration logic is triggered, the file system client interacts with the metadata server to verify the metadata. After verification, the data is temporarily stored in the cache acceleration module. After the cache acceleration module completes the idle memory block suitability check, it can batch schedule data to the target storage for persistence. If the cache acceleration module is out of memory or misses the cache acceleration marker, the data falls into the system page cache, which is then written back to the target storage. When processing read requests, the file system client first queries the cache acceleration module and the system page cache. If the cache is hit, the data is returned directly. If it is not hit, the data is read from the target storage and cached in the cache acceleration module and the system page cache, realizing efficient operation and cache acceleration of the data read and write process.

[0040] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0041] Figure 2A flowchart of a data processing method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, an execution subject of the embodiment of the present application is a file system client shown in FIG. 2, which can also be other devices with similar functions, and the present application is not limited in this regard. A detailed description of the method is as follows. Figure 2 Figure 1

[0042] S201: receiving a write request sent by an application layer for to-be-written data, and judging whether a transmission mode of the write request is a direct read-write mode.

[0043] Specifically, the file system client listens to I / O operations initiated by the application layer in real time, and when a write request for specific data is captured, the request header or protocol field is parsed preferentially, such as the flag bit of the open function in the Linux system or the mode identifier of the custom communication protocol. It is identified whether the write operation is a "direct read-write mode", such as O_DIRECT flag. If it is a direct read-write mode, it will bypass the cache and directly access the storage, and if it is not a direct read-write mode, it will continue to the subsequent judgment.

[0044] S202: if the transmission mode of the write request is not a direct read-write mode, it is further judged whether a client cache mechanism is enabled for the file system client.

[0045] Specifically, after excluding the direct read-write mode, the file system client needs to check its own configuration or runtime state by reading a preset configuration file, such as the cct_enable flag of the Lustre distributed file system client or a global variable in the memory, which is used to record whether the cache mechanism is activated, to judge whether the client cache mechanism is enabled. If it is enabled, it enters the cache acceleration process; if it is not enabled, it goes through the traditional page cache write path.

[0046] The process of enabling the client cache mechanism is as follows: the process starts with executing the lctl cct enable instruction, which is trapped into the kernel through the ioctl system call, triggering the ll_dir_ioctl operation; then memory block pre-application is performed, the cache acceleration module applies a batch of 16K memory blocks according to 50% of the system memory and puts them into the idle memory block list; then a flush thread is started, which is used to persist the cache data subsequently; finally, it is marked that the client cache mechanism has been enabled, and the enabling process of the client cache mechanism function is completed. The ioctl is a system call function for device control in the Linux system.

[0047] ​​When it is necessary to close the client cache mechanism, the process of closing the client cache mechanism is as follows: the process starts from executing the lctl cct disable instruction, which falls into the kernel through an ioctl system call and triggers the ll_dir_ioctl operation; then the client cache mechanism is marked as closed, and new written data no longer enters the cache layer. At the same time, the flushing module performs the operation of flushing data, and after the data flushing is completed, the flushing thread automatically exits, thereby releasing the memory block previously applied for the client cache mechanism, and finally completing the closing process of the client cache mechanism function.

[0048] S203: If the client cache mechanism is enabled, it is further judged whether the metadata of the to-be-written data or the parent directory of the to-be-written data carries the cache acceleration mark.

[0049] Specifically, the lctl cct enable instruction is first executed, which falls into the kernel through an ioctl system call and triggers the ll_file_ioctl operation; then it is judged whether the file carries the cache acceleration mark, if the file carries the cache acceleration mark, then directly return -EALREADY, if the file does not carry the cache acceleration mark, the file system client sends a setattr request to the metadata server, the metadata server receives the request, acquires the file distributed lock, adds the cache acceleration mark in the i-flags of the file inode and persists, then releases the lock and replies to the file system client; the file system client receives the reply, updates the local file metadata, and finally returns success, completing the process. The detailed description of the judgment process is as follows:

[0050] Sa1: reading the metadata of the to-be-written data, wherein the metadata includes a cache acceleration state field.

[0051] Specifically, the client initiates a metadata query request to the metadata server, carrying the file path or inode number of the to-be-written data, and extracts the field containing the "cache acceleration state" from the metadata structure stored in the metadata server, such as the custom accel_flag identifier.

[0052] Sa2: judging whether the cache acceleration state field matches the cache acceleration mark.

[0053] Specifically, the read metadata field is compared with the preset cache acceleration mark, such as a specific binary value or an enumeration type, if the field value is completely consistent, such as accel_flag=1 corresponding to the mark activation, then it is preliminarily determined that the match is matched.

[0054] Sa3: if the cache acceleration state field of the to-be-written data matches the cache acceleration mark, it is determined that the metadata of the to-be-written data carries the cache acceleration mark.

[0055] Specifically, when the field value completely matches the tag, directly tagging the file itself allows cache acceleration, and subsequent client cache mechanism processes are given priority.

[0056] Sa4: If the cache acceleration status field of the to-be-written data does not match the cache acceleration tag, metadata of a direct parent directory to which the to-be-written data belongs is obtained, and it is determined whether the cache acceleration tag is carried in the metadata of the direct parent directory.

[0057] Specifically, if the file itself is not tagged with cache acceleration, the file system client recursively queries the metadata of the parent directory, and can traverse upwards through the file path, such as for the file path: / data / checkpoint / file, the file metadata is queried first, and then the checkpoint directory metadata is queried, and the checking logic steps of Sa1-Sa2 are repeated to check the cache acceleration tag at the directory level.

[0058] Sa5: If the cache acceleration tag is not carried in the metadata of the direct parent directory, the metadata of the parent directory at the next level is read and determined, until the root directory is traversed.

[0059] Specifically, the directory hierarchy of the file system is followed, such as the root directory being / , the parent directory metadata is queried layer by layer upwards, to ensure that all possible levels of inherited acceleration tags are covered, such as when the parent directory is tagged with acceleration, the child file inherits by default.

[0060] Sa6: If the cache acceleration tag is matched in the metadata of any level of parent directory, it is determined that the cache acceleration tag is carried in the metadata of the parent directory of the to-be-written data.

[0061] Specifically, as long as the metadata of a directory at a certain level contains the cache acceleration tag, it is determined that the file can inherit the acceleration policy, and the file system client cache process is triggered.

[0062] Sa7: If the cache acceleration tag is not matched in the metadata of the to-be-written data and all level parent directory metadata, it is determined that the cache acceleration tag is not carried in the metadata of the to-be-written data and the parent directory of the to-be-written data.

[0063] Specifically, when the file itself and all upper level directories do not have acceleration tags, cache acceleration is not enabled by default, and subsequent traditional write processes are followed, and the page cache is directly written.

[0064] Specifically, if the cache acceleration tag is not carried in the metadata of the to-be-written data and the parent directory of the to-be-written data, the following steps are performed:

[0065] Sb1: Send an update request for the metadata of the to-be-written data to the metadata server, so that the metadata server writes the cache acceleration tag in the metadata of the to-be-written data and performs persistent storage.

[0066] Sb2: Receive a successful mark write response from the metadata server and complete adding a cache acceleration mark to the metadata of the data to be written.

[0067] Specifically, if a file or directory is detected without an acceleration flag, the client can proactively initiate a metadata update request to the metadata server, such as update_metadata(file,accel_flag=1), to write the acceleration flag to the file metadata and persist it. After the metadata server returns a "flag written successfully" response, subsequent write operations can trigger cache acceleration.

[0068] S204: If the metadata of the data to be written or the parent directory of the data to be written carries a cache acceleration flag, the size of the free memory block of the preset cache is obtained.

[0069] Specifically, the client cache mechanism maintains a list of memory blocks, such as pre-allocated continuous / discrete memory areas, and calculates the total capacity of all free memory blocks or the single maximum available capacity by traversing the list and querying the status identifier of the memory block to prepare for subsequent data storage.

[0070] S205: Determine whether the size of the free memory block is greater than or equal to the size of the data to be written, and whether the data to be written does not overlap with the data in the preset cache.

[0071] Specifically, the judgment process is described in detail as follows:

[0072] Sc1: Traverse the memory block list of the preset cache, filter out the memory blocks in the idle state, and count the actual available capacity of each idle memory block.

[0073] Specifically, the memory block list of the preset cache is traversed, the memory blocks in the idle state are screened out, and the actual available capacity of each idle memory block is counted.

[0074] Sc2: Compare the total actual available capacity of the filtered idle memory blocks with the size of the data to be written to determine whether there is an idle memory block with a capacity that meets the requirement.

[0075] Specifically, the size of the data to be written is obtained through the length parameter of the write call, and is compared one by one with the available capacity of the free memory blocks to determine whether there is at least one memory block with a capacity greater than or equal to the data size, ensuring that the data can be stored completely.

[0076] Sc3: If there is a free memory block with a capacity that meets the requirement, determine whether the size of the free memory block is greater than or equal to the size of the data to be written.

[0077] Specifically, when a memory block meeting the capacity requirement is found, the memory block is marked as "pre-occupied" and enters the data overlap checking link.

[0078] Sc4: Compare the file identifier and data range information corresponding to the data to be written in the target storage with the file identifier and data range information corresponding to the data in the pre-set cache, and determine whether there is an identical file identifier and an intersection of data ranges.

[0079] Specifically, the client cache mechanism maintains an unpersisted data index table, which records the file path to which the data temporarily stored in the cache belongs, the offset and length of the data in the file. The data to be written needs to check whether it overlaps with the existing cached data of the same file. For example, if the offset 100-200 of file A is cached, and the new offset 150-250 is written, it will overlap.

[0080] Sc5: If there is an identical file identifier and an intersection of data ranges, it is determined that the data to be written overlaps with the data in the pre-set cache.

[0081] Specifically, when it is detected that the file is the same and the offset interval has an intersection, it is determined that the data overlaps, and the cache needs to be avoided to cause data coverage or inconsistency. At this time, the memory block needs to be skipped.

[0082] S206: If the size of the idle memory block is greater than or equal to the size of the data to be written, and the data to be written does not overlap with the data in the pre-set cache, save the data to be written to the pre-allocated memory block of the pre-set cache.

[0083] Specifically, the client directly writes the data of the application layer into the client cache mechanism and the allocated idle memory block through a memory copy function, and updates the state of the memory block to used, records the file to which the data stored in the memory block belongs, the offset, and other metadata.

[0084] S207: Traverse the memory block list of the pre-set cache, and save the data to be written to the target storage.

[0085] Specifically, the client starts a flush thread or a timing task, traverses the memory blocks pre-allocated by the client cache mechanism, collects the data contents of these memory blocks, and calls the data write function of the target storage to batch persist the data to the target storage.

[0086] In conclusion, through multi-level judgment and precise cache control, the data write performance can be significantly improved. First, the transmission mode and cache mechanism are screened to avoid invalid cache operations. Then, the cache acceleration mark is used for hierarchical verification to accurately locate the data that needs to be accelerated and reduce resource waste. At the same time, relying on the memory block management of the preset cache, combined with the idle size and data overlap check, the cache safety is guaranteed, and the distributed lock dependence is reduced through batch persistence, greatly reducing the write latency and adapting to the demand for hundred-microsecond-level latency in high-performance computing scenarios.

[0087] In another embodiment provided in the application, for a read request of an application layer, the processing flow of the file system client is as follows:

[0088] S301: receiving a read request sent by an application layer for target data, and judging whether the transmission mode of the read request is a direct read-write mode.

[0089] Specifically, the file system client continuously monitors the I / O request of the application layer. When receiving a read request for target data, the mode identifier in the request is parsed, such as the O_DIRECT flag of the open function in the Linux system or the transmission mode field of the custom protocol. If the identifier is a direct read-write mode, the cache process is directly skipped, and the data is read from the target storage; if it is a non-direct read-write mode, the subsequent cache mechanism verification link is entered to ensure that the cache acceleration logic only acts on appropriate read request scenarios.

[0090] S302: If the transmission mode of the read request is not a direct read-write mode, verify whether the client cache mechanism of the file system client is enabled.

[0091] Specifically, after confirming the non-direct read-write mode, the client checks the local configuration parameters, such as the cache switch state variable, the cct_enabled field in the configuration file, or the kernel state global flag, to determine whether the client cache mechanism is in the enabled state. If it is not enabled, the traditional read process is directly executed to read from the target storage or system page cache; if it is enabled, the target data is further verified to see if it meets the cache acceleration condition to avoid invalid processing for scenarios where the cache is not turned on.

[0092] S303: If the client cache mechanism is enabled, continue to verify whether the cache acceleration mark is carried in the metadata of the target data or the parent directory of the target data.

[0093] Specifically, the client sends a metadata query request to the metadata server, carrying the file path of the target data, reads the cache acceleration status field in its metadata, such as accel_flag. If the target data itself carries the metadata, it is directly determined to meet the acceleration condition; if not, recursively query the metadata of its immediate parent directory, grandparent directory, and so on until the root directory. As long as the metadata of any level of directory exists the mark, it is determined that the target data can inherit the cache acceleration strategy.

[0094] S304: If the metadata of the target data or the parent directory of the target data carries the cache acceleration mark, query whether the target data exists in the preset cache.

[0095] Specifically, the client cache mechanism maintains an index table recording the file identification, data range and corresponding memory block address of the cached data. When it is confirmed that the target data meets the cache acceleration condition, the client compares the file identification and data range of the target data by searching the index table to determine whether the complete target data exists in the preset cache, such as the offset and length of the cached data completely covering the required range of the read request.

[0096] S305: If the target data exists in the preset cache, read the target data from the preset cache and return to the application layer.

[0097] Specifically, when it is found that the target data exists in the preset cache, the file system client copies the data in the cache to the user-mode buffer of the application layer through memory address mapping, and returns a read success response to the application layer.

[0098] S306: If the target data does not exist in the preset cache, read the target data from the target storage, and save the target data to the pre-allocated memory block in the preset cache.

[0099] Specifically, if the target data is not hit in the cache, the file system client reads the target data from the target storage through the access interface of the target storage, transmits the data to the application layer, and triggers the cache write logic: checks the list of pre-allocated memory blocks in the preset cache, filters out the idle memory blocks that match the size and have no data overlap, writes the read target data to the memory block, and updates the cache index table, and marks the memory block as "used". Subsequently, if the application layer reads the data again, it can directly hit the pre-allocated cache, reducing the overhead of repeated access to the target storage.

[0100] In summary, through multi-level judgment such as transmission mode screening, cache mechanism verification, and acceleration mark verification, accurate delivery of cache resources is realized, and cache acceleration is enabled only for non-direct read-write requests that meet the conditions; at the same time, with the help of hit query and data pre-storage mechanism of the preset cache, the cached data can be directly read from the memory block, and the data that does not hit is stored in the cache in time after reading, which greatly reduces repeated access to the target storage, reduces the dependence on distributed locks and cross-node communication latency, significantly improves the response speed of read requests, and adapts to the demand of low latency and high throughput of storage systems in artificial intelligence and high-performance computing scenarios.

[0101] Figure 3 A structural diagram of a data processing apparatus provided by an embodiment of the present application is shown in FIG. 1. As shown in the figure, the data processing apparatus comprises a transmission mode judgment module 301, a cache mechanism enabling judgment module 302, a cache acceleration mark judgment module 303, an idle memory size obtaining module 304, an idle memory availability judgment module 305, a data storage module 306, and a persistent storage module 307. Figure 3

[0102] The transmission mode judgment module 301 is configured to receive a write request sent by an application layer for to-be-written data, and judge whether the transmission mode of the write request is a direct read-write mode.

[0103] The cache mechanism enabling judgment module 302 is configured to, if the transmission mode of the write request is not the direct read-write mode, continue to judge whether a client cache mechanism is enabled by a file system client.

[0104] The cache acceleration mark judgment module 303 is configured to, if the client cache mechanism is enabled, continue to judge whether a cache acceleration mark is carried in metadata of the to-be-written data or a parent directory of the to-be-written data.

[0105] The idle memory size obtaining module 304 is configured to, if the cache acceleration mark is carried in the metadata of the to-be-written data or the parent directory of the to-be-written data, obtain the size of an idle memory block of a preset cache.

[0106] The idle memory availability judgment module 305 is configured to judge whether the size of the idle memory block is greater than or equal to the size of the to-be-written data, and whether the to-be-written data and the data in the preset cache have no overlap.

[0107] The data storage module 306 is configured to, if the size of the idle memory block is greater than or equal to the size of the to-be-written data, and the to-be-written data and the data in the preset cache have no overlap, save the to-be-written data into a memory block pre-allocated by the preset cache.

[0108] The persistent storage module 307 is configured to traverse a memory block list of the preset cache, and save the to-be-written data into a target storage.

[0109] ​In a possible implementation, the cache acceleration mark judgment module 303 is specifically configured to read the metadata of the to-be-written data, where the metadata includes a cache acceleration state field; determine whether the cache acceleration state field matches the cache acceleration mark; if the cache acceleration state field of the to-be-written data matches the cache acceleration mark, it is determined that the cache acceleration mark is carried in the metadata of the to-be-written data; if the cache acceleration state field of the to-be-written data does not match the cache acceleration mark, the metadata of the immediate parent directory to which the to-be-written data belongs is obtained, and it is determined whether the cache acceleration mark is carried in the metadata of the immediate parent directory; if the cache acceleration mark is not carried in the metadata of the immediate parent directory, the metadata of the parent directory at the next level is read and determined, until the root directory is traversed; if the cache acceleration mark is matched in the metadata of the parent directory at any level, it is determined that the cache acceleration mark is carried in the metadata of the parent directory of the to-be-written data; if the cache acceleration mark is not matched in the metadata of the to-be-written data and the metadata of all levels of parent directories, it is determined that the cache acceleration mark is not carried in the metadata of the to-be-written data and the parent directory of the to-be-written data.

[0110] In a possible implementation, the apparatus further includes a cache acceleration mark adding module, which is configured to send, to the metadata server, an update request for the metadata of the to-be-written data, so that the metadata server writes the cache acceleration mark in the metadata of the to-be-written data and performs persistent storage; and receive a mark writing success response returned by the metadata server, to complete the addition of the cache acceleration mark in the metadata of the to-be-written data.

[0111] In a possible implementation, the idle memory available judgment module 305 is specifically configured to traverse a memory block list of a preset cache, filter out memory blocks in an idle state, and count actual available capacities of the memory blocks in the idle state; compare the sum of the actual available capacities of the filtered memory blocks in the idle state with the size of the to-be-written data, to determine whether there is an idle memory block with a capacity meeting the requirement; if there is an idle memory block with a capacity meeting the requirement, it is determined that the size of the idle memory block is greater than or equal to the size of the to-be-written data; compare the file identifier and the data range information corresponding to the data in the preset cache that have not completed target storage persistence with the file identifier and the data range information of the to-be-written data, to determine whether there is the same file identifier and an intersection of data ranges; if there is the same file identifier and an intersection of data ranges, it is determined that the to-be-written data overlaps with the data in the preset cache.

[0112] In a possible implementation, the apparatus further includes a read request processing module, configured to receive a read request sent by an application layer for target data, determine whether a transmission mode of the read request is a direct read-write mode, if the transmission mode of the read request is not the direct read-write mode, check whether a client cache mechanism is enabled for the file system client, if the client cache mechanism is enabled, continue to check whether a cache acceleration flag is carried in metadata of the target data or a parent directory of the target data, if the cache acceleration flag is carried in the metadata of the target data or the parent directory of the target data, query whether the target data exists in a preset cache, if the target data exists in the preset cache, read the target data from the preset cache and return the target data to the application layer, and if the target data does not exist in the preset cache, read the target data from a target storage, and save the target data into a memory block pre-allocated in the preset cache.

[0113] The data processing apparatus provided in this embodiment can execute the method provided in the method embodiments, and has similar implementation principles and technical effects. Details are not described herein again.

[0114] Figure 4 FIG. 1 shows a structural diagram of a file system client provided in this embodiment. As shown in FIG. 1, the file system client provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the file system client further includes a communication component 403. The processor 401, the memory 402, and the communication component 403 are connected through a bus 404. Figure 4

[0115] In the implementation process, the at least one processor 401 executes computer execution instructions stored in the memory 402, so that the at least one processor 401 executes the method described above.

[0116] The specific implementation process of the processor 401 can refer to the method embodiments described above, and has similar implementation principles and technical effects. Details are not described herein again.

[0117] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor. ​

[0118] The memory can include a Random Access Memory (RAM) and can also include a Non-volatile Memory (NVM), such as at least one disk memory.

[0119] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0120] The embodiment of the present application further provides a computer program product, comprising a computer program, which is executed by a processor to implement the method described above.

[0121] The embodiment of the present application further provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0122] The readable storage medium described above can be realized by any type of volatile or non-volatile storage device or their combination, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0123] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0124] The division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0126] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0127] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0128] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes the steps including the above-mentioned method embodiments when executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0129] It should be understood that many of the materials and devices exemplified in this disclosure are articles of manufacture (i.e., articles of manufacture) according to this disclosure. The articles of manufacture can be manufactured as such or can be manufactured by combining the materials and devices exemplified in this disclosure. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be understood that, in some embodiments, equivalents to the specific electrode structures and / or methods described herein can be employed without departing from the scope of the application. Accordingly, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized into," and variations thereof herein, is meant to encompass the items listed thereafter, and equivalents thereof as well as additional items. Although the foregoing application has been described in some detail by way of illustration and example, it is not to be limited thereby, but rather, only by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: Applicable to file system clients, including: receiving a write request sent by the application layer for data to be written, and determining whether the transmission mode of the write request is a direct read-write mode; If the transmission mode of the write request is not the direct read-write mode, then continue to determine whether the file system client has enabled a client cache mechanism; If the client cache mechanism is enabled, continue to determine whether the metadata of the data to be written or the parent directory of the data to be written carries a cache acceleration flag; If the metadata of the data to be written or the parent directory of the data to be written carries the cache acceleration flag, obtaining the size of the free memory block of the preset cache; Determine whether the size of the free memory block is greater than or equal to the size of the data to be written, and whether the data to be written does not overlap with the data in the preset cache; If the size of the free memory block is greater than or equal to the size of the data to be written, and the data to be written does not overlap with the data in the preset cache, then the data to be written is saved in the memory block pre-allocated in the preset cache; Traverse the memory block list of the preset cache and save the data to be written to the target storage.

2. The method according to claim 1, characterized in that The determining whether the metadata of the data to be written or the parent directory of the data to be written carries a cache acceleration flag includes: Reading metadata of the data to be written, wherein the metadata includes a cache acceleration status field; Determining whether the cache acceleration status field matches the cache acceleration flag; If the cache acceleration status field of the data to be written matches the cache acceleration flag, determining that the metadata of the data to be written carries the cache acceleration flag; If the cache acceleration status field of the data to be written does not match the cache acceleration flag, obtaining metadata of the direct parent directory to which the data to be written belongs, and determining whether the metadata of the direct parent directory carries the cache acceleration flag; If the metadata of the direct parent directory does not carry the cache acceleration flag, continue to read and judge the metadata of the parent directory of the next level up until traversing to the root directory; If the cache acceleration flag is matched in the metadata of the parent directory at any level, it is determined that the metadata of the parent directory of the data to be written carries the cache acceleration flag; If the cache acceleration flag is not matched in the metadata of the data to be written and the metadata of all hierarchical parent directories, it is determined that the cache acceleration flag is not carried in the metadata of the data to be written and the parent directories of the data to be written.

3. The method according to claim 2, characterized in that After determining that neither the data to be written nor the metadata of the parent directory of the data to be written carries the cache acceleration flag, the method further includes: Sending an update request for the metadata of the data to be written to a metadata server, so that the metadata server writes the cache acceleration mark into the metadata of the data to be written and performs persistent storage; A mark write success response returned by the metadata server is received, and the addition of the cache acceleration mark in the metadata of the data to be written is completed.

4. The method according to claim 1, wherein The determining whether the size of the free memory block is greater than or equal to the size of the data to be written, and the data to be written does not overlap with the data in the preset cache, includes: Traversing the memory block list of the preset cache, filtering out the memory blocks in the idle state, and counting the actual available capacity of each idle memory block; Comparing the total actual available capacity of the screened idle memory blocks with the size of the data to be written to determine whether there is an idle memory block with a capacity that meets the requirement; If there is a free memory block with a capacity that meets the requirement, determining that the size of the free memory block is greater than or equal to the size of the data to be written; Compare the file identifier and data range information corresponding to the data in the preset cache that has not completed the target storage persistence with the file identifier and data range information of the data to be written to determine whether there is an identical file identifier and an intersection in the data range; If the same file identifier exists and the data ranges have an intersection, it is determined that the data to be written overlaps with the data in the preset cache.

5. The method according to claim 1, wherein Also includes: receiving a read request sent by the application layer for target data, and determining whether a transmission mode of the read request is a direct read-write mode; If the transmission mode of the read request is not the direct read-write mode, then checking whether the file system client has enabled a client cache mechanism; If the client cache mechanism is enabled, then continue to check whether the metadata of the target data or the parent directory of the target data carries a cache acceleration flag; If the metadata of the target data or the parent directory of the target data carries a cache acceleration flag, querying whether the target data exists in the preset cache; If the target data exists in the preset cache, reading the target data from the preset cache and returning it to the application layer; If the target data does not exist in the preset cache, the target data is read from the target storage and saved in a memory block pre-allocated in the preset cache.

6. A data processing device, characterized in that: Applicable to file system clients, including: A transmission mode determination module is configured to receive a write request sent by the application layer for data to be written, and determine whether the transmission mode of the write request is a direct read-write mode; a cache mechanism activation judgment module, configured to, if the transmission mode of the write request is not the direct read-write mode, continue to judge whether the file system client has activated the client cache mechanism; A cache acceleration mark judgment module is used to, if the client cache mechanism is enabled, continue to judge whether the metadata of the data to be written or the parent directory of the data to be written carries a cache acceleration mark; a free memory size acquisition module, configured to acquire the size of a free memory block of a preset cache if the metadata of the data to be written or the parent directory of the data to be written carries the cache acceleration flag; a free memory available judgment module, configured to judge whether the size of the free memory block is greater than or equal to the size of the data to be written, and whether the data to be written does not overlap with the data in the preset cache; a data storage module, configured to save the data to be written into a memory block pre-allocated in the preset cache if the size of the free memory block is greater than or equal to the size of the data to be written and the data to be written does not overlap with the data in the preset cache; The persistent storage module is used to traverse the memory block list of the preset cache and save the data to be written to the target storage.

7. The device according to claim 6, characterized in that The cache acceleration mark judgment module, Specifically used for reading metadata of the data to be written, wherein the metadata includes a cache acceleration status field; judging whether the cache acceleration status field matches the cache acceleration tag; if the cache acceleration status field of the data to be written matches the cache acceleration tag, then determining that the metadata of the data to be written carries the cache acceleration tag; If the cache acceleration status field of the data to be written does not match the cache acceleration flag, obtaining metadata of the direct parent directory to which the data to be written belongs, and determining whether the metadata of the direct parent directory carries the cache acceleration flag; If the metadata of the direct parent directory does not carry the cache acceleration flag, continue to read and judge the metadata of the parent directory of the next level up until traversing to the root directory; If the cache acceleration flag is matched in the metadata of the parent directory at any level, it is determined that the metadata of the parent directory of the data to be written carries the cache acceleration flag; If the cache acceleration flag is not matched in the metadata of the data to be written and the metadata of all hierarchical parent directories, it is determined that the cache acceleration flag is not carried in the metadata of the data to be written and the parent directories of the data to be written.

8. A file system client, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 5 when executed by a processor.

Images