Data processing method, electronic device, and storage medium

Abstract

This application provides a data processing method, an electronic device, and a storage medium. The method includes: a processing module receiving a first request from a target application, the first request requesting to read data from a first file; the first file's size is less than a preset value; the processing module determining whether a cache file of the first file exists on the client's cache disk; if the cache file exists, the processing module determining whether the cache file is valid; if the cache file is valid, the processing module reading the cache file from the client's cache disk to obtain the data of the first file; and the processing module sending the data of the first file to the target application. This method improves the efficiency of reading massive amounts of small file data.

Description

Technical Field

[0001] This application relates to the field of data reading, and more particularly to a data processing method, electronic device, and storage medium. Background Technology

[0002] Lustre is a high-performance distributed file system commonly used in large computing clusters and high-performance computing (HPC) environments. In Lustre, metadata and data are stored and managed separately. Data is stored across multiple object storage objects (OSTs) in Lustre. This storage approach supports storing extremely large files and provides high-speed concurrent read / write performance.

[0003] However, in scenarios such as model training, it is necessary to read a large number of small files. The size of these small files is generally smaller than the bandwidth of a single I / O (Input / Output) access. Frequent I / O data reading will waste network bandwidth resources and result in poor transmission performance of the Lustre file system.

[0004] Therefore, there is a problem of low efficiency in reading massive amounts of small files. Summary of the Invention

[0005] This application provides a data processing method, an electronic device, and a storage medium to solve the technical problem of low efficiency in reading massive amounts of small files.

[0006] In a first aspect, this application provides a data processing method, comprising: a processing module receiving a first request from a target application, the first request being for requesting to read data from a first file; the file size of the first file being less than a preset value;

[0007] The processing module determines whether the cache file of the first file exists on the client's cache disk;

[0008] If a cached file for the first file is determined to exist, the processing module determines whether the cached file is valid.

[0009] If the cache file of the first file is determined to be valid, the processing module reads the cache file of the first file from the client's cache disk to obtain the data of the first file;

[0010] The processing module sends the data of the first file to the target application.

[0011] Optionally, as described above, the first request includes an identifier of the first file, and the processing module determines whether a cache file of the first file exists on the client's cache disk, including:

[0012] The processing module searches for the metadata information of the first file based on the identifier of the first file;

[0013] If the metadata information of the first file contains the cache path of the first file on the client's cache disk, then it is determined that the cache file of the first file exists on the client's cache disk.

[0014] Optionally, as described above, the processing module determines whether the cache file of the first file is valid, including:

[0015] The processing module obtains the latest modification time of the first file on the client's cache disk and the latest modification time of the first file on the server from the metadata information of the first file;

[0016] If the latest modification time of the first file on the client's cache disk is the same as the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is valid.

[0017] Optionally, as described above, the processing module reads the cache file of the first file from the client's cache disk to obtain the data of the first file, including:

[0018] The processing module obtains the cache path of the first file on the client's cache disk from the metadata information of the first file;

[0019] The processing module opens the cache file of the first file on the client's cache disk based on the cache path;

[0020] The processing module obtains the handle of the cached file and reads the cached file of the first file from the client's cache disk based on the handle of the cached file to obtain the data of the first file.

[0021] Optionally, as described above, the processing module determines whether a cache file of the first file exists on the client's cache disk, including:

[0022] The processing module determines whether the first request carries a first flag, which is used to indicate a write operation on the data of the first file;

[0023] If it is determined that the first request does not carry the first tag, the processing module determines whether the cache file of the first file exists on the client's cache disk.

[0024] Optionally, as described above, the method further includes:

[0025] The processing module receives a second request from the target application, which requests the closure of the first file;

[0026] The processing module determines whether the cache file of the first file is open on the client's cache disk;

[0027] If it is determined that the cache file of the first file is already open on the client's cache disk, the processing module deletes the handle of the cache file and closes the cache file.

[0028] Optionally, as described above, the method further includes:

[0029] If the preset conditions are met, the processing module obtains the file path of the first file on the server from the metadata information of the first file;

[0030] The processing module retrieves the data of the first file from the server based on the file path of the first file on the server.

[0031] The preset conditions include at least one of the following:

[0032] The cache file for the first file does not exist on the client's cache disk;

[0033] The cache file for the first file is invalid;

[0034] The first request carries the first tag.

[0035] Optionally, as described above, the client also includes a cache management module, and the method further includes:

[0036] The cache management module receives a third request from the cache management terminal. The third request includes file directory information, which includes the identifier of the file directory and the identifiers of multiple files under the file directory. The third request is used to add the data of multiple files under the directory to the client's cache disk in advance.

[0037] The cache management module retrieves data from multiple files in the file directory from the server based on the file directory information and caches the data of multiple files to the client's cache disk.

[0038] Optionally, as described above, the method further includes:

[0039] When the processing module determines that the latest modification time of the first file on the client's cache disk is inconsistent with the latest modification time of the first file on the server, the processing module sends an indication message to the cache management module.

[0040] Based on the instructions, the cache management module shall perform at least one of the following:

[0041] Delete the cached file of the first file from the client's cache disk;

[0042] Retrieve the latest data of the first file from the server and cache the latest data of the first file to the client's cache disk.

[0043] Optionally, as described above, the client's cache disk includes a RAM disk and a non-volatile memory standard NVMe disk. The method further includes:

[0044] When the storage space on the RAM disk is insufficient, the cache management module caches file data to the NVMe disk.

[0045] Secondly, this application provides a data processing apparatus, comprising:

[0046] The processing module is used to receive a first request from the target application, which requests to read data from a first file; the file size of the first file is less than a preset value.

[0047] The processing module is also used to determine whether the cache file of the first file exists on the client's cache disk.

[0048] The processing module is also used to determine whether the cache file is valid if a cache file of the first file is found to exist.

[0049] The processing module is also used to read the cache file of the first file from the client's cache disk to obtain the data of the first file if it is determined that the cache file of the first file is valid.

[0050] The processing module is also used to send the data of the first file to the target application.

[0051] The cache management module is used to receive a third request from the cache management terminal. The third request includes file directory information, which includes the identifier of the file directory and the identifiers of multiple files in the file directory. The third request is used to add the data of multiple files in the directory to the client's cache disk in advance.

[0052] The cache management module is also used to retrieve data of multiple files in a file directory from the server based on the file directory information, and cache the data of multiple files to the client's cache disk.

[0053] Thirdly, this application provides an electronic device, comprising:

[0054] The processor, and the memory that is in communication with the processor;

[0055] The memory stores the instructions that the computer executes;

[0056] The processor executes computer execution instructions stored in memory to implement the method of any of the first aspects of this application.

[0057] Fourthly, this application provides a computer-readable storage medium, comprising:

[0058] The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method of the first aspect of this application.

[0059] Fifthly, this application provides a computer program product, comprising:

[0060] A computer program that, when executed by a processor, implements the method of any of the first aspects of this application.

[0061] The data processing method, electronic device, and storage medium provided in this application receive a first request from a target application through a processing module. The first request requests to read data from a first file. The file size of the first file is less than a preset value. The processing module determines whether a cached file of the first file exists on the client's cache disk. If a cached file of the first file exists, the processing module determines whether the cached file is valid. If the cached file of the first file is valid, the processing module reads the cached file of the first file from the client's cache disk to obtain the data of the first file. The processing module then sends the data of the first file to the target application. The method of this application improves the efficiency of reading massive amounts of small file data. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 An architecture diagram of the Lustre file system provided in this application;

[0064] Figure 2 A schematic diagram of the architecture of the data processing method provided in this application;

[0065] Figure 3 A flowchart illustrating a data processing method provided in this application;

[0066] Figure 4 A flowchart illustrating another data processing method provided in this application;

[0067] Figure 5 A flowchart illustrating another data processing method provided in this application;

[0068] Figure 6 A flowchart illustrating yet another data processing method provided in this application;

[0069] Figure 7 A schematic diagram of the data processing apparatus provided in this application;

[0070] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0071] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0074] First, let me explain the terms used in this application:

[0075] Lustre is a high-performance distributed file system suitable for computing environments that require high performance and large-scale data processing capabilities. Lustre is a portmanteau of Linux and Cluster. The Lustre file system was developed by Cluster File Systems Inc., founded by Peter Braam. In this application, the Lustre file system can also be simply referred to as Lustre.

[0076] Data is a symbolic record of objective things. It is a formalized representation of information that can be further processed and interpreted, in order to facilitate the communication, interpretation, or processing of data.

[0077] Metadata is data that describes data, or structured data that provides information about a resource. It mainly describes data attributes and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0078] Figure 1 The architecture diagram of the Lustre file system provided in this application is as follows: Figure 1As shown, the Lustre file system mainly includes: at least one Lustre client, one Metadata Server (MDS), and at least one Object Storage Server (OSS).

[0079] Among them, MDS is responsible for managing metadata information, managing the unified namespace of the file system, and allowing clients to directly access metadata information. The metadata object (Metadata Target, MDT) associated with MDS is the physical storage device or logical storage unit that stores metadata, which can be understood as the storage backend of MDS.

[0080] OSS manages object storage objects (OSTs). Multiple OSTs can be mounted under each OSS. OSTs are used to store file data (i.e., data).

[0081] The client runs the Lustre file system and interacts with MDS and OSS. It implements POSIX file system interfaces such as open(), read(), and write(), thereby effectively enabling communication with users.

[0082] When a user needs to read data from the Lustre file system, the client first obtains the metadata information of the target data through the MDS, and then reads the target data from the OST through the MDS.

[0083] In the Lustre file system, Lustre file data can be striped and stored on multiple different OSTs. Striping storage can support Lustre storage of very large files and provide high-speed concurrent read and write performance.

[0084] However, in scenarios such as model training, where it is necessary to read a large number of small files, the size of which is generally smaller than the bandwidth of a single I / O access, frequent and large-scale access to the OST will waste network bandwidth resources and result in poor transmission performance of the Lustre file system.

[0085] Therefore, there is a problem of low efficiency in reading massive amounts of small file data.

[0086] The data processing method provided in this application is intended to solve the above-mentioned technical problems of the prior art.

[0087] The data processing method provided in this application pre-adds the data of multiple files in a directory to the client's cache disk through a cache management module. When reading a large amount of small file data, the processing module receives a first request from the target application, which requests to read the data of a first file; it then determines whether a cache file of the first file exists on the client's cache disk; if a cache file of the first file exists, it determines whether the cache file is valid; if the cache file of the first file is valid, it reads the cache file of the first file from the client's cache disk to obtain the data of the first file. The data processing method provided in this application conveniently and quickly improves the efficiency of reading large amounts of small file data.

[0088] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0089] Figure 2 A schematic diagram of the architecture of the data processing method provided in this application is shown below. Figure 2 As shown, the architecture of the data processing method consists of three parts: client, server, and cache management.

[0090] The client has the target application, processing module, cache disk, and cache management module installed.

[0091] The target application can refer to the target application that needs to read a large number of small files, such as the target application related to model training. Small files can refer to files whose size is less than the bandwidth of a single I / O access, such as a 64kb file.

[0092] The processing module receives requests from the target application to read data from the first file and sends the data from the first file to the target application. The processing module stores the file directory and metadata information for the files within that directory.

[0093] The cache disk serves as the client's local storage disk, used to cache file data.

[0094] The cache management module is used to manage the cache disk. When the cache management terminal sends a request to the cache management module to add or update data, the cache management module will perform the operation of adding or updating data on the cache disk.

[0095] The data source for the cache management module to add or update data is the server. The server stores a large amount of complete data, and all data is distributed and stored in multiple OSTs. If the processing module cannot find the latest data, it can also obtain it from the server.

[0096] The cache management system is typically managed by staff.

[0097] To better demonstrate the data interaction between the various implementing entities Figure 3 A flowchart illustrating a data processing method provided in this application is shown below. Figure 3 As shown, the method includes:

[0098] S301. The target application sends a first request to the processing module. The first request is used to request to read data from the first file. The file size of the first file is less than a preset value.

[0099] Accordingly, the processing module receives the first request from the target application.

[0100] The first file can refer to a file that the target application needs to read in order to obtain the data in the first file.

[0101] The first request includes the identifier of the first file, which can be the file name.

[0102] The preset value is used to indicate whether the first file is a small file. A small file can refer to a file whose size is less than the bandwidth of a single I / O access. For example, the preset value can be set to 64kb.

[0103] S302. The processing module determines whether the cache file of the first file exists on the client's cache disk.

[0104] Before the processing module determines whether the cache file of the first file exists on the client's cache disk, the method may also include:

[0105] S302a, The processing module determines whether the first request carries a first flag, which is used to indicate that a write operation is performed on the data of the first file.

[0106] For example, the first flag can be a write flag. When there is a write flag, it can be determined that the first file can be read and written. When there is no write flag, it can be determined that the first file is read-only.

[0107] S302b If it is determined that the first request does not carry the first tag, the processing module determines whether the cache disk of the client contains a cache file of the first file.

[0108] In one alternative implementation, the processing module is deployed with a Lustre file system client, so the processing module can directly obtain the file names in the unified namespace and the corresponding metadata information of the files.

[0109] In one alternative implementation, determining whether a cache file of the first file exists on the client's cache disk may include the following steps:

[0110] S3021. The processing module searches for the metadata information of the first file based on the identifier of the first file.

[0111] The metadata information of the first file describes the data attributes of the first file. The metadata information of the first file may include at least one of the following: the identifier of the first file, the size of the first file, the cache path of the data of the first file on the client, the file path of the data of the first file on the server, and the latest modification time of the first file on the server.

[0112] S3022. If the metadata information of the first file contains the cache path of the first file on the client's cache disk, then it is determined that the cache file of the first file exists on the client's cache disk.

[0113] Accordingly, if the metadata information of the first file does not contain the cache path of the first file on the client's cache disk, it is assumed that the cache file of the first file does not exist on the client's cache disk.

[0114] S303. If it is determined that a cache file for the first file exists, the processing module determines whether the cache file is valid.

[0115] In one alternative implementation, the processing module determines whether the cached file is valid, and the steps may include:

[0116] S3031. The processing module obtains the latest modification time of the first file on the client's cache disk and the latest modification time of the first file on the server from the metadata information of the first file.

[0117] For example, if the cache management module caches the data of the first file to the client's cache disk, the latest modification time of the first file in the client's cache disk is consistent with the latest modification time of the first file on the server, for example, the latest modification time is 10:00 on December 30, 2024.

[0118] If the data of the first file on the server changes at 10:30 on December 30, 2024, then the latest modification time of the first file on the server will be 10:30 on December 30, 2024, while the latest modification time of the first file on the client's cache disk will still be 10:00 on December 30, 2024. At this time, the latest modification time of the first file on the client's cache disk is inconsistent with the latest modification time of the first file on the server.

[0119] S3032. If the latest modification time of the first file on the client's cache disk is consistent with the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is valid.

[0120] Accordingly, if the latest modification time of the first file on the client's cache disk is inconsistent with the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is invalid.

[0121] In one optional implementation, when the processing module determines that the latest modification time of the first file on the client's cache disk is inconsistent with the latest modification time of the first file on the server, the processing module sends an indication message to the cache management module.

[0122] Based on the instructions, the cache management module shall perform at least one of the following:

[0123] S3032a. Delete the cache file of the first file from the client's cache disk.

[0124] S3032b: Obtain the latest data of the first file from the server and cache the latest data of the first file to the client's cache disk.

[0125] The instruction information includes the identifier of the first file and the cache path of the cached file of the first file on the client's cache disk.

[0126] For example, if the cache management module needs to delete the data of the first file from the client's cache disk, the parameters in the execution statement can be {'subdir':' / whml','file_path':' / tt3','cache_type':'1'}, where subdir indicates that the cache file of the first file is located in the client's directory / whml, file_path indicates that the cache file of the first file is located in the client's path / tt3, and a cache_type value of 1 indicates that the data of the cache file of the first file is cached on the client's RAM disk, and a cache_type value of 2 indicates that the data of the cache file of the first file is cached on the client's NVMe disk.

[0127] S304. If it is determined that the cache file of the first file is valid, the processing module reads the cache file of the first file from the client's cache disk to obtain the data of the first file.

[0128] In one alternative implementation, S304 may include:

[0129] S3041. The processing module obtains the cache path of the first file on the client's cache disk from the metadata information of the first file.

[0130] S3042. The processing module opens the cache file of the first file on the client's cache disk according to the cache path.

[0131] For example, the statement to open a file can be open().

[0132] S3043. The processing module obtains the handle of the cache file and reads the cache file of the first file from the cache disk of the client according to the handle of the cache file to obtain the data of the first file.

[0133] In computer programming, a file handle is an abstract pointer used to identify and access a file.

[0134] For example, the handle can be stored in a structure, which can be named "struct file". In the Linux kernel, "struct file" is a data structure that represents an open file. Whenever a file is opened, the kernel creates an instance of "struct file" to represent that open file.

[0135] S305, The processing module sends the data of the first file to the target application.

[0136] The data processing method provided in this application can directly read the data of the first file from the cache file in the cache disk after determining that the cache file of the first file exists and that the cache file of the first file is valid, without having to read from the OST corresponding to the Lustre file system, thus conveniently and quickly improving the efficiency of reading massive amounts of small file data.

[0137] In the fields of computer science and software engineering, a complete process of reading file data includes opening the file, reading the file, and closing the file to ensure the integrity of the resource. Figure 4 A flowchart illustrating another data processing method provided in this application, in one optional implementation, such as... Figure 4 As shown, after the processing module sends the data of the first file to the target application, the method further includes:

[0138] S401. The target application sends a second request to the processing module, the second request being used to request the closure of the first file.

[0139] Accordingly, the processing module receives a second request from the target application.

[0140] S402, The processing module determines whether the cache file of the first file on the client's cache disk has been opened.

[0141] S403. If it is determined that the cache file of the first file is open on the client's cache disk, the processing module deletes the handle of the cache file and closes the cache file.

[0142] For example, as described above Figure 3 The handles of cached files are stored in the structure "struct file". Deleting the handle of a cached file means removing the handle of that cached file from the structure "struct file".

[0143] For example, the statement to close the cache file can be close().

[0144] The alternative data processing method provided in this application ensures the integrity of the file reading process and avoids the error-proneness of frequent file reading.

[0145] Figure 5 A flowchart illustrating another data processing method provided in this application is shown below. Figure 5 As shown, the method includes:

[0146] S501. If the preset conditions are met, the processing module obtains the file path of the first file on the server from the metadata information of the first file.

[0147] S502. The processing module obtains the data of the first file from the server according to the file path of the first file on the server.

[0148] The preset conditions include at least one of the following:

[0149] a. The cache file for the first file does not exist on the client's cache disk;

[0150] b. The cache file for the first file is invalid;

[0151] c. The first request carries a first tag.

[0152] Specifically, determining that the cache file of the first file does not exist on the client's cache disk can be derived from S3022 above; determining that the cache file of the first file is invalid can be derived from S3032 above; and determining that the first request carries the first tag can be derived from S302a above. These will not be elaborated further here.

[0153] The data processing method provided in this application takes into account the requirement of obtaining data from the first file from the server under preset conditions, thus ensuring the supportability of the source of the read data.

[0154] To meet the target application's need to frequently read massive amounts of small files, pre-caching these small files on a cache disk can improve the efficiency of the processing module in reading such data. Figure 6 A flowchart illustrating another data processing method provided in this application is shown below. Figure 6 As shown, the method includes:

[0155] S601, The cache management terminal sends a third request to the cache management module. The third request includes file directory information, which includes the identifier of the file directory and the identifiers of multiple files under the file directory. The third request is used to add the data of multiple files under the directory to the client's cache disk in advance.

[0156] Correspondingly, the cache management module receives a third request from the cache management terminal.

[0157] Among them, the file directory information can help staff identify specific file directories based on actual application scenarios. In one possible implementation, the file directory information can also be obtained through intelligent strategy filtering. For example, a neural network model can take the historical cached file directory information in the client's cache disk as input and output the commonly used file directory information in actual application scenarios.

[0158] For example, practical application scenarios include, but are not limited to, model training scenarios. In practical application scenarios, there is a need to frequently read a large number of small files, which are contained in at least one directory.

[0159] S602 The cache management module retrieves data from multiple files in the file directory from the server based on the file directory information, and caches the data of multiple files to the client's cache disk.

[0160] The cache disk includes a memory disk and a non-volatile memory standard NVME disk. In one optional implementation, when the storage space of the memory disk is insufficient, the cache management module caches the file data to the NVME disk.

[0161] In one possible implementation, the cache management module adds a directory to the cache disk on the client based on the file directory information. For example, the parameter in the execution statement can be {'cache_path':' / mnt / mem_cache','cache_type':1}, where cache_path in the execution statement indicates that the cache disk path is ' / mnt / mem_cache', and cache_type in the execution statement is 1 to indicate that it is a memory disk, or 2 to indicate that it is an NVME disk.

[0162] For example, if it is necessary to query cache disk information before adding a directory, a GET request in computer language can be used to obtain the response content for querying cache disk information. This response content could be, for example, {'nvme_cache_used':0,'mem_cache used':1295163392,'nvme cache size':2147483648,'mem cache':' / mnt / memcache','mem cache size':2147483648,'nvme_cache':' / mnt / nvme_cache'}, where nvme_cache_used indicates that the cache disk is an NVMe disk and the used bytes are 0, mem_cache used indicates that the cache disk is a RAM disk and the used bytes are 1295163392, nvme cache size indicates that the NVMe disk's capacity is 2147483648 bytes, and mem cache... The size indicates that the capacity of the RAM disk is 2147483648 bytes, the path of the RAM disk is set to / mnt / mem cache, and the path of the NVMe disk is set to / mnt / nvme_cache.

[0163] For example, if it is necessary to cache the data of multiple files in a file directory to the client's cache disk, the parameters in the execution statement can be {'subdir':' / whml','file_path':' / tt3','cache_type':'1'}, where subdir represents the client's directory as / whml, file_path represents the file path of the files in this directory on the client as / tt3, and a cache_type value of 1 indicates that the data of multiple files in this directory obtained by the server is cached to the client's RAM disk, and a cache_type value of 2 indicates that the data of multiple files in this directory obtained by the server is cached to the client's NVMe disk.

[0164] In one possible implementation, the client can have more than one NVMe disk.

[0165] This application provides another data processing method that allows the client's cache management module to pre-add file data containing file directory information to the client's cache disk. This enables the processing module to directly read from the client's cache disk upon receiving the first request from the target application, instead of reading from the OST corresponding to the Lustre file system. This significantly improves the efficiency of reading massive amounts of small file data. Furthermore, the client's cache management module manages the cache disk independently of the processing module, providing the processing module with more thread resources for reading massive amounts of small file data and further improving its reading efficiency.

[0166] Figure 7 A schematic diagram of the data processing apparatus provided in this application is shown below. Figure 7 As shown, the data processing apparatus 70 provided in this application includes: a processing module 701 and a cache management module 702;

[0167] Processing module 701 is used to receive a first request from the target application, the first request being to request the reading of data from a first file; the file size of the first file is less than a preset value.

[0168] The processing module 701 is also used to determine whether the cache file of the first file exists on the client's cache disk.

[0169] The processing module 701 is also used to determine whether the cache file is valid if it is determined that a cache file of the first file exists.

[0170] The processing module 701 is also used to read the cache file of the first file from the cache disk of the client to obtain the data of the first file if it is determined that the cache file of the first file is valid.

[0171] The processing module 701 is also used to send the data of the first file to the target application.

[0172] In one possible implementation, the processing module 701 is further configured to: locate the metadata information of the first file based on the identifier of the first file;

[0173] If the metadata information of the first file contains the cache path of the first file on the client's cache disk, then it is determined that the cache file of the first file exists on the client's cache disk.

[0174] In one possible implementation, the processing module 701 is further configured to: obtain the latest modification time of the first file on the client's cache disk and the latest modification time of the first file on the server from the metadata information of the first file;

[0175] If the latest modification time of the first file on the client's cache disk is the same as the latest modification time of the first file on the server, then the cache file of the first file is determined to be valid.

[0176] In one possible implementation, the processing module 701 is further configured to:

[0177] Obtain the cache path of the first file on the client's cache disk from the metadata information of the first file;

[0178] Based on the cache path, open the cache file of the first file on the client's cache disk;

[0179] Obtain the handle of the cached file, and read the cached file of the first file from the client's cache disk based on the handle of the cached file to obtain the data of the first file.

[0180] In one possible implementation, the processing module 701 is further configured to: determine whether the first request carries a first flag, the first flag being used to indicate a write operation on the data of the first file;

[0181] If it is determined that the first request does not carry the first tag, determine whether the cache file of the first file exists on the client's cache disk.

[0182] In one possible implementation, the processing module 701 is further configured to: receive a second request from the target application, the second request being for requesting the closure of the first file;

[0183] Determine if the cache file for the first file is open on the client's cache disk;

[0184] If it is determined that the cache file of the first file is open on the client's cache disk, delete the handle of the cache file and close the cache file.

[0185] In one possible implementation, the processing module 701 is further configured to: if a preset condition is met, obtain the file path of the first file on the server from the metadata information of the first file;

[0186] Based on the file path of the first file on the server, retrieve the data of the first file from the server;

[0187] The preset conditions include at least one of the following:

[0188] The cache file for the first file does not exist on the client's cache disk;

[0189] The cache file for the first file is invalid;

[0190] The first request carries the first tag.

[0191] The cache management module 702 is used to receive a third request from the cache management terminal. The third request includes file directory information, which includes the identifier of the file directory and the identifiers of multiple files under the file directory. The third request is used to add the data of multiple files under the directory to the client's cache disk in advance.

[0192] Based on the file directory information, retrieve the data of multiple files in the file directory from the server and cache the data of multiple files to the client's cache disk.

[0193] The cache management module 702 is also used to send an instruction message to the cache management module when the processing module determines that the latest modification time of the first file on the client's cache disk is inconsistent with the latest modification time of the first file on the server.

[0194] Based on the instructions, the cache management module shall perform at least one of the following:

[0195] Delete the cached file of the first file from the client's cache disk;

[0196] Retrieve the latest data of the first file from the server and cache the latest data of the first file to the client's cache disk.

[0197] The cache management module 702 is also used to cache file data to the NVME disk when the storage space of the memory disk is insufficient.

[0198] Figure 8 A schematic diagram of the structure of the electronic device provided in this application, such as... Figure 8 As shown, the electronic device 80 provided in this application embodiment includes at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.

[0199] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.

[0200] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0201] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0202] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0203] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0204] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0205] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0206] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0207] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0208] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0209] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0210] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0211] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0212] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0213] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0214] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0215] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a client, the client including a processing module and a target application, the processing module deploying a client with the Lustre file system, the method includes: The processing module receives a first request from the target application, the first request being for requesting to read data from a first file; the file size of the first file is less than a preset value. The processing module determines whether the cache file of the first file exists on the client's cache disk; If it is determined that a cached file of the first file exists, the processing module obtains the latest modification time of the first file on the client's cache disk and the latest modification time of the first file on the server from the metadata information of the first file; If the latest modification time of the first file in the cache disk of the client is the same as the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is valid. If the cache file of the first file is determined to be valid, the processing module reads the cache file of the first file from the cache disk of the client to obtain the data of the first file; The processing module sends the data of the first file to the target application; The client also includes a cache management module, and the method further includes: The cache management module receives a third request from the cache management terminal. The third request includes file directory information, which includes the identifier of the file directory and the identifiers of multiple files under the file directory. The third request is used to add the data of multiple files under the directory to the client's cache disk in advance. The cache management module obtains data of multiple files in the file directory from the server based on the file directory information, and caches the data of the multiple files to the client's cache disk; The client's cache disk includes a memory disk and a non-volatile memory standard NVME disk. The method further includes: when the storage space of the memory disk is insufficient, the cache management module caches file data to the NVME disk.

2. The method according to claim 1, characterized in that, The first request includes an identifier for the first file, and the processing module determines whether a cache file of the first file exists on the client's cache disk, including: The processing module searches for the metadata information of the first file based on the identifier of the first file; If the metadata information of the first file contains the cache path of the first file on the client's cache disk, then it is determined that the cache file of the first file exists on the client's cache disk.

3. The method according to claim 1 or 2, characterized in that, The processing module reads the cache file of the first file from the client's cache disk to obtain the data of the first file, including: The processing module obtains the cache path of the first file on the client's cache disk from the metadata information of the first file; The processing module opens the cache file of the first file on the client's cache disk according to the cache path; The processing module obtains the handle of the cached file and reads the cached file of the first file from the cache disk of the client according to the handle of the cached file to obtain the data of the first file.

4. The method according to any one of claims 1 to 3, characterized in that, The processing module determines whether a cache file of the first file exists on the client's cache disk, including: The processing module determines whether the first request carries a first flag, which is used to indicate a write operation on the data of the first file; If it is determined that the first request does not carry the first tag, the processing module determines whether the cache disk of the client contains a cache file of the first file.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The processing module receives a second request from the target application, the second request being used to request the closure of the first file; The processing module determines whether the cache file of the first file on the client's cache disk is open; If it is determined that the cache file of the first file is open on the cache disk of the client, the processing module deletes the handle of the cache file and closes the cache file.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the preset conditions are met, the processing module obtains the file path of the first file on the server from the metadata information of the first file; The processing module obtains the data of the first file from the server according to the file path of the first file on the server. The preset conditions include at least one of the following: The cache disk of the client does not contain a cache file of the first file; The cache file for the first file is invalid; The first request carries a first tag.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: When the processing module determines that the latest modification time of the first file in the cache disk of the client is inconsistent with the latest modification time of the first file on the server, the processing module sends an indication message to the cache management module; The cache management module performs at least one of the following actions based on the instruction information: Delete the cache file of the first file from the cache disk of the client; The latest data of the first file is obtained from the server, and the latest data of the first file is cached to the cache disk of the client.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images