Data processing method, electronic equipment and storage medium

Abstract

The invention provides a data processing method, electronic equipment and a storage medium. The method comprises the steps that a processing module receives a first request from a target application program, wherein the first request is used for requesting to read data of a first file; the file size of the first file is smaller than a preset value; the processing module determines whether a cache file of the first file exists in a cache disk of the client; if it is determined that the cache file of the first file exists, the processing module determines whether the cache file is valid or not; if it is determined that the cache file of the first file is valid, the processing module reads the cache file of the first file from a cache disk of the client to obtain data of the first file; and the processing module sends the data of the first file to the target application program. According to the method provided by the invention, the efficiency of reading the data of massive small files is improved.

Description

Technical Field

[0001] The present application relates to the field of data reading, and in particular to a data processing method, an electronic device and a storage medium. Background Art

[0002] Lustre is a high-performance distributed file system that is commonly used in large computing clusters and high-performance computing (HPC) environments. In the Lustre file system, metadata and data are stored and managed separately. Data is stored in multiple object storage targets (OSTs) of Lustre. This storage method can support Lustre to store very large files and provide high-speed concurrent read and write performance.

[0003] However, in scenarios such as model training, it is necessary to read a large number of small files. The size of small files is generally smaller than the bandwidth of a single I / O (Input / Output) access. Frequent reading of data through I / O will cause a waste of network bandwidth resources and lead to poor transmission performance of the Lustre file system.

[0004] Therefore, there is a problem of low data efficiency in reading massive small files. Summary of the invention

[0005] The present application provides a data processing method, an electronic device and a storage medium, which are used to solve the technical problem of low data efficiency in reading a large number of small files.

[0006] In a first aspect, the present application provides a data processing method, comprising: a processing module receiving a first request from a target application, the first request being used to request to read data of a first file; a file size of the first file being smaller than a preset value;

[0007] The processing module determines whether a cache file of the first file exists in the cache disk of the client;

[0008] If it is determined that there is a cache file of the first file, the processing module determines whether the cache file is valid;

[0009] If it is determined that the cache file of the first file is valid, the processing module reads the cache file of the first file from the cache disk of the client to obtain data of the first file;

[0010] The processing module sends the data of the first file to the target application.

[0011] Optionally, as in the above method, the first request includes an identifier of the first file, and the processing module determines whether a cache file of the first file exists in the cache disk of the client, including:

[0012] The processing module searches for metadata information of the first file according to the identifier of the first file;

[0013] If the metadata information of the first file includes a cache path of the first file on the cache disk of the client, it is determined that a cache file of the first file exists on the cache disk of the client.

[0014] Optionally, as in the above method, the processing module determines whether the cache file of the first file is valid, including:

[0015] The processing module obtains the latest modification time of the first file on the cache disk of the client and the latest modification time of the first file on the server from the metadata information of the first file;

[0016] If the latest modification time of the first file on the cache disk of the client is consistent with the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is valid.

[0017] Optionally, as in the above method, the processing module reads the cache file of the first file from the cache disk of the client to obtain the data of the first file, including:

[0018] The processing module obtains a cache path of the first file on a cache disk of the client from metadata information of the first file;

[0019] The processing module opens the cache file of the first file in the cache disk of the client according to the cache path;

[0020] The processing module obtains a handle of the cache file, and reads the cache file of the first file from the cache disk of the client according to the handle of the cache file to obtain data of the first file.

[0021] Optionally, as in the above method, the processing module determines whether there is a cache file of the first file in the cache disk of the client, including:

[0022] The processing module determines whether the first request carries a first mark, where the first mark is used to indicate a write operation to be performed on data of the first file;

[0023] If it is determined that the first request does not carry the first mark, the processing module determines whether a cache file of the first file exists in the cache disk of the client.

[0024] Optionally, as in the above method, the method further comprises:

[0025] The processing module receives a second request from the target application, where the second request is used to request closing the first file;

[0026] The processing module determines whether a cache file of the first file in the cache disk of the client has been opened;

[0027] If it is determined that the cache file of the first file in the cache disk of the client has been opened, the processing module deletes the handle of the cache file and closes the cache file.

[0028] Optionally, as in the above method, the method further comprises:

[0029] If the preset condition is met, the processing module obtains the file path of the first file on the server from the metadata information of the first file;

[0030] The processing module obtains data of the first file from the server according to the file path of the first file on the server;

[0031] The preconditions include at least one of the following:

[0032] The cache file of the first file does not exist in the cache disk of the client;

[0033] The cache file of the first file is invalid;

[0034] The first request carries a first tag.

[0035] Optionally, as in the above method, the client further includes a cache management module, and the method further includes:

[0036] The cache management module receives a third request from the cache management terminal, the third request including file directory information, the file directory information including an identifier of the file directory and identifiers of multiple files under the file directory; the third request is used to add data of the multiple files under the directory to the cache disk of the client in advance;

[0037] The cache management module obtains data of multiple files in the file directory from the server according to the file directory information, and caches the data of the multiple files to the cache disk of the client.

[0038] Optionally, as in the above method, the method further comprises:

[0039] When the processing module determines that the latest modification time of the first file on the cache disk of the client is inconsistent with the latest modification time of the first file on the server, the processing module sends indication information to the cache management module;

[0040] The cache management module performs at least one of the following according to the instruction information:

[0041] Deleting a cache file of the first file in the cache disk of the client from the cache disk of the client;

[0042] The latest data of the first file is obtained from the server, and the latest data of the first file is cached to the cache disk of the client.

[0043] Optionally, as in the above method, the cache disk of the client includes a memory disk and a non-volatile memory standard NVME disk, and the method further includes:

[0044] When the storage space of the memory disk is insufficient, the cache management module caches the file data to the NVME disk.

[0045] In a second aspect, the present application provides a data processing device, including:

[0046] The processing module is used to receive a first request from a target application, where the first request is used to request to read data of a first file; the file size of the first file is smaller than a preset value.

[0047] The processing module is further used to determine whether there is a cache file of the first file in the cache disk of the client.

[0048] The processing module is further configured to determine whether the cache file is valid if it is determined that there is a cache file of the first file.

[0049] The processing module is further configured to read the cache file of the first file from the cache disk of the client to obtain data of the first file if it is determined that the cache file of the first file is valid.

[0050] The processing module is also used to send the data of the first file to the target application.

[0051] The cache management module is used to receive a third request from the cache management terminal, the third request includes file directory information, the file directory information includes an identifier of the file directory and identifiers of multiple files under the file directory; the third request is used to add data of multiple files under the directory to the cache disk of the client in advance;

[0052] The cache management module is also used to obtain data of multiple files in the file directory from the server according to the file directory information, and cache the data of the multiple files to the cache disk of the client.

[0053] In a third aspect, the present application provides an electronic device, including:

[0054] A processor, and a memory communicatively connected to the processor;

[0055] Memory stores computer-executable instructions;

[0056] The processor executes the computer-executable instructions stored in the memory to implement any method of the first aspect of the present application.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, comprising:

[0058] The computer-readable storage medium stores computer-executable instructions, which are used to implement the method of the first aspect of the present application when executed by a processor.

[0059] In a fifth aspect, the present application provides a computer program product, including:

[0060] A computer program which, when executed by a processor, implements any one of the methods of the first aspect of the present application.

[0061] The data processing method, electronic device and storage medium provided by the present application receive a first request from a target application through a processing module, the first request is used to request to read the data of a first file; the file size of the first file is less than a preset value; the processing module determines whether there is a cache file of the first file on the cache disk of the client; if it is determined that there is a cache file of the first file, the processing module determines whether the cache file is valid; if it is determined that the cache file of the first file is valid, the processing module reads the cache file of the first file from the cache disk of the client to obtain the data of the first file; the processing module sends the data of the first file to the target application. The method of the present application improves the efficiency of reading data of a large number of small files. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0063] Figure 1 The architecture diagram of the Lustre file system provided for this application;

[0064] Figure 2 A schematic diagram of the architecture of the data processing method provided in this application;

[0065] Figure 3 A flowchart of a data processing method provided in this application;

[0066] Figure 4 A flowchart of another data processing method provided for this application;

[0067] Figure 5 A flowchart of another data processing method provided in this application;

[0068] Figure 6 A flowchart of another data processing method provided by the present application;

[0069] Figure 7 A schematic diagram of the structure of the data processing device provided by this application;

[0070] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0071] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0074] First, the terms involved in this application are explained:

[0075] Lustre is a high-performance distributed file system suitable for computing environments that require high performance and large-scale data processing capabilities. Lustre is a portmanteau of Linux and Cluster. The Lustre file system was developed by Cluster File Systems Inc., founded by Peter Braam. In this application, the Lustre file system can also be referred to as Lustre.

[0076] Data is a symbolic record of objective things and a formal representation of information that can be reprocessed and interpreted to suit the communication, interpretation or processing of data.

[0077] Metadata is data that describes data, or structured data used to provide information about a resource. It is mainly information that describes data attributes and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0078] Figure 1 The architecture diagram of the Lustre file system provided for this application is as follows Figure 1As shown, the Lustre file system mainly includes: at least one Lustre client (client), a metadata server (Metadata Server, MDS) and at least one object storage service (Object Storage Server, OSS).

[0079] Among them, MDS is responsible for managing metadata information, managing the unified namespace of the file system, and allowing clients to directly access metadata information. The metadata object (Metadata Target, MDT) associated with MDS is a physical storage device or logical storage unit for storing metadata, which can be understood as the storage backend of MDS.

[0080] OSS manages object storage objects (OSTs). Multiple OSTs can be mounted under each OSS. OSTs are used to store file data (i.e. data).

[0081] The Client runs the Lustre file system and interacts with the MDS and OSS. It implements the POSIX file system interface, such as open(), read(), and write(), to effectively communicate with users.

[0082] When a user needs to read data from the Lustre file system, the client first obtains the metadata information of the target data through the MDS, and then reads the target data from the OST through the MDS.

[0083] In the Lustre file system, Lustre file data can be stored in stripes on multiple different OSTs. The striped storage method can support Lustre to store very large files and provide high-speed concurrent read and write performance.

[0084] However, when it is necessary to read a large amount of small file data, such as in a model training scenario, the size of the data is generally smaller than the bandwidth of a single I / O access. Frequent and large-scale access to OST will cause a waste of network bandwidth resources and lead to poor transmission performance of the Lustre file system.

[0085] Therefore, there is a problem of low efficiency in reading massive amounts of small file data.

[0086] The data processing method provided in this application is intended to solve the above technical problems in the prior art.

[0087] The data processing method provided by the present application adds the data of multiple files under a directory to the cache disk of the client in advance through the cache management module. When in the scenario of reading the data of a large number of small files, the processing module receives a first request from the target application, and the first request is used to request to read the data of the first file; determines whether there is a cache file of the first file on the cache disk of the client; if it is determined that there is a cache file of the first file, determines whether the cache file is valid; if it is determined that the cache file of the first file is valid, reads the cache file of the first file from the cache disk of the client to obtain the data of the first file. The data processing method provided by the present application conveniently and quickly improves the efficiency of reading the data of a large number of small files.

[0088] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0089] Figure 2 A schematic diagram of the architecture of the data processing method provided in this application, such as Figure 2 As shown, the architecture of the data processing method includes three parts: client, server, and cache management;

[0090] The client is installed with a target application, a processing module, a cache disk, and a cache management module.

[0091] The target application may refer to a target application that needs to read a large number of small files, such as a target application related to model training. The small file may refer to a file whose size is smaller than the bandwidth of a single I / O access, such as a 64kb file.

[0092] The processing module is used to receive a request from a target application to read data of a first file, and send the data of the first file to the target application. The processing module stores the file directory and metadata information of the files under the file directory.

[0093] The cache disk is a local storage disk of the client and is used to cache file data.

[0094] The cache management module is used to manage the cache disk. When the cache management terminal sends a request to add data or update data to the cache management module, the cache management module adds data or updates data to the cache disk.

[0095] Among them, the data source for the cache management module to add or update data is the server, which stores a large amount of complete data. All data is distributed and stored in multiple OSTs. If the processing module does not find the latest data, it can also be obtained from the server.

[0096] The cache management end is generally managed by staff.

[0097] In order to better display the data interaction between various execution entities, Figure 3 A flow chart of a data processing method provided in this application is as follows: Figure 3 As shown, the method includes:

[0098] S301. A target application sends a first request to a processing module, where the first request is used to request to read data of a first file; and a file size of the first file is smaller than a preset value.

[0099] Accordingly, the processing module receives a first request from the target application.

[0100] The first file may refer to a file that the target application needs to read to obtain data of the first file.

[0101] The first request includes an identifier of the first file, and the identifier of the first file may be a file name.

[0102] The preset value is used to indicate whether the first file is a small file. A small file may refer to a file whose size is smaller than the bandwidth of a single I / O access. Exemplarily, the preset value may be set to 64 kb.

[0103] S302: The processing module determines whether a cache file of the first file exists in the cache disk of the client.

[0104] Before the processing module determines whether a cache file of the first file exists in the cache disk of the client, the method may further include:

[0105] S302a: The processing module determines whether the first request carries a first tag, where the first tag is used to indicate a write operation to be performed on data of the first file.

[0106] Exemplarily, the first mark may be a write mark. When there is a write mark, it can be determined that the first file can be read and written. When there is no write mark, it can be determined that the first file is read-only.

[0107] S302b: If it is determined that the first request does not carry the first tag, the processing module determines whether there is a cache file of the first file in the cache disk of the client.

[0108] In an optional implementation, the processing module is deployed with a Lustre file system client, so the processing module can directly obtain the file name of the unified namespace and metadata information corresponding to the file.

[0109] In an optional implementation, the step of determining whether a cache file of the first file exists in the cache disk of the client may include:

[0110] S3021. The processing module searches for metadata information of the first file according to an identifier of the first file.

[0111] Among them, the metadata information of the first file is information that describes the data attributes of the first file. The metadata information of the first file may include at least one of the identifier of the first file, the size of the first file, the cache path of the data of the first file on the client, the file path of the data of the first file on the server, and the latest modification time of the first file on the server.

[0112] S3022: If the metadata information of the first file includes a cache path of the first file on the cache disk of the client, it is determined that a cache file of the first file exists on the cache disk of the client.

[0113] Correspondingly, if the metadata information of the first file does not include the cache path of the first file in the cache disk of the client, it is considered that the cache file of the first file does not exist in the cache disk of the client.

[0114] S303: If it is determined that there is a cache file of the first file, the processing module determines whether the cache file is valid.

[0115] In an optional implementation, the processing module determines whether the cache file is valid, and the steps may include:

[0116] S3031. The processing module obtains the latest modification time of the first file on the cache disk of the client and the latest modification time of the first file on the server from the metadata information of the first file.

[0117] Exemplarily, if the cache management module caches the data of the first file to the cache disk of the client, the latest modification time of the first file on the cache disk of the client is consistent with the latest modification time of the first file on the server, for example, the latest modification time is 10:00 on December 30, 2024;

[0118] When the data of the first file on the server is changed at 10:30 on December 30, 2024, the latest modification time of the first file on the server is 10:30 on December 30, 2024, while the latest modification time of the first file on the cache disk of the client is still 10:00 on December 30, 2024. At this time, the latest modification time of the first file on the cache disk of the client is inconsistent with the latest modification time of the first file on the server.

[0119] S3032: If the latest modification time of the first file on the cache disk of the client is consistent with the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is valid.

[0120] Correspondingly, if the latest modification time of the first file on the cache disk of the client is inconsistent with the latest modification time of the first file on the server, the processing module determines that the cache file of the first file is invalid.

[0121] In an optional implementation, when the processing module determines that the latest modification time of the first file on the cache disk of the client is inconsistent with the latest modification time of the first file on the server, the processing module sends an indication message to the cache management module;

[0122] The cache management module performs at least one of the following according to the instruction information:

[0123] S3032a. Delete the cache file of the first file in the cache disk of the client from the cache disk of the client.

[0124] S3032b. Obtain the latest data of the first file from the server, and cache the latest data of the first file to the cache disk of the client.

[0125] The indication information includes an identifier of the first file and a cache path of a cache file of the first file in a cache disk of the client.

[0126] Exemplarily, if the cache management module needs to delete the data of the first file from the cache disk of the client, the parameters in the execution statement can be {'subdir':' / whml','file_path':' / tt3','cache_type':'1'}, wherein subdir indicates that the directory of the cache file of the first file on the client is / whml, file_path indicates that the path of the cache file of the first file on the client is / tt3, and a cache_type value of 1 indicates that the data of the cache file of the first file is cached in the client's memory disk. If the cache_type value is 2, it indicates that the data of the cache file of the first file is cached in the client's NVME disk.

[0127] S304: If it is determined that the cache file of the first file is valid, the processing module reads the cache file of the first file from the cache disk of the client to obtain data of the first file.

[0128] In an optional implementation, S304 may include:

[0129] S3041. The processing module obtains a cache path of the first file on a cache disk of the client from metadata information of the first file.

[0130] S3042: The processing module opens the cache file of the first file in the cache disk of the client according to the cache path.

[0131] Exemplarily, the execution statement for opening a file may be open().

[0132] S3043: The processing module obtains a handle of the cache file, and reads the cache file of the first file from the cache disk of the client according to the handle of the cache file to obtain data of the first file.

[0133] Among them, in computer programming, a file handle is an abstract pointer used to identify and access a file.

[0134] For example, the handle may be stored in a structure, and the name of the structure may be "struct file". In the Linux kernel, "struct file" is a data structure that represents an open file. Whenever a file is opened, the kernel creates an instance of "struct file" to represent the open file.

[0135] S305: The processing module sends the data of the first file to the target application.

[0136] The data processing method provided by the present application directly reads the data of the first file from the cache file in the cache disk after determining that the cache file of the first file exists and that the cache file of the first file is valid, without the need to read from the OST corresponding to the Lustre file system, thereby conveniently and quickly improving the efficiency of reading massive small file data.

[0137] In the field of computer science and software engineering, a complete step of reading file data includes opening the file, reading the file, and closing the file to achieve resource integrity. Figure 4 A flow chart of another data processing method provided by the present application, in an optional implementation manner, as Figure 4 As shown, after the processing module sends the data of the first file to the target application, the method further includes:

[0138] S401. The target application sends a second request to the processing module, where the second request is used to request to close the first file.

[0139] Accordingly, the processing module receives a second request from the target application.

[0140] S402: The processing module determines whether a cache file of the first file in the cache disk of the client has been opened.

[0141] S403: If it is determined that the cache file of the first file in the cache disk of the client has been opened, the processing module deletes the handle of the cache file and closes the cache file.

[0142] For example, as mentioned above Figure 3 The handle of the cache file is stored in the structure "struct file", and deleting the handle of the cache file can be removing the handle of the cache file in the structure "struct file".

[0143] Exemplarily, the execution statement for closing the cache file may be close().

[0144] Another data processing method provided by the present application makes the file reading process complete and avoids the error-proneness of frequent file reading.

[0145] Figure 5 A flowchart of another data processing method provided in this application is shown as follows: Figure 5 As shown, the method includes:

[0146] S501: If a preset condition is met, the processing module obtains a file path of the first file on the server from metadata information of the first file.

[0147] S502: The processing module obtains data of the first file from the server according to the file path of the first file on the server.

[0148] The preconditions include at least one of the following:

[0149] a. The cache file of the first file does not exist in the cache disk of the client;

[0150] b. The cache file of the first file is invalid;

[0151] c. The first request carries a first tag.

[0152] Among them, determining that the cache file of the first file does not exist in the client's cache disk can be derived from the above S3022, determining that the cache file of the first file is invalid can be derived from the above S3032, and determining that the first request carries the first mark can be derived from the above S302a, which will not be repeated here.

[0153] Another data processing method provided by the present application takes into account obtaining the data of the first file from the server when preset conditions are met, thereby ensuring the supportability of the source of the read data.

[0154] In order to meet the needs of the target application to frequently read the data of a large number of small files, the cache disk pre-caches the data of a large number of small files to improve the efficiency of the processing module in reading the data of small files. Figure 6 A flowchart of another data processing method provided by this application is as follows: Figure 6 As shown, the method includes:

[0155] S601. The cache management end sends a third request to the cache management module. The third request includes file directory information. The file directory information includes an identifier of the file directory and identifiers of multiple files under the file directory. The third request is used to add data of multiple files under the directory to the client's cache disk in advance.

[0156] Correspondingly, the cache management module receives the third request from the cache management end.

[0157] Among them, the file directory information can be used by staff to confirm specific file directories based on actual application scenarios. In a possible implementation method, the file directory information can also be obtained by intelligent strategy screening. For example, the neural network model uses the historical cache file directory information in the client's cache disk as input, and outputs the file directory information commonly used in actual application scenarios.

[0158] Exemplarily, the actual application scenario includes but is not limited to a model training scenario. In the actual application scenario, there is a need to frequently read a large number of small files, wherein the large number of small files are contained in at least one directory.

[0159] S602: The cache management module obtains data of multiple files in the file directory from the server according to the file directory information, and caches the data of the multiple files to the cache disk of the client.

[0160] The cache disk includes a memory disk and a non-volatile memory standard NVME disk. In an optional implementation, when the storage space of the memory disk is insufficient, the cache management module caches the file data to the NVME disk.

[0161] In one possible implementation, the cache management module adds a directory to the cache disk of the client according to the file directory information. Exemplarily, the parameters in the execution statement may be {'cache_path':' / mnt / mem_cache', 'cache_type':1}, where the cache_path in the execution statement indicates that the cache disk path is ' / mnt / mem_cache', and the cache_type in the execution statement is 1, indicating a memory disk, and if cache_type is 2, it indicates an NVME disk.

[0162] For example, if you need to query the cache disk information before adding a directory, you can use a computer language get request to obtain the response content of the cache disk information query, and the response content can be, for example, {'nvme_cache_used':0,'mem_cache used':1295163392,'nvme cache size':2147483648,'mem cache':' / mnt / memcache','mem cache size':2147483648,'nvme_cache':' / mnt / nvme_cache'}, where nvme_cache_used indicates that the cache disk is an NVME disk and the used bytes are 0, mem_cache used indicates that the cache disk is a memory disk and the used bytes are 1295163392, nvme cache size indicates that the capacity of the NVME disk is 2147483648 bytes, and mem cache size indicates that the capacity of the memory disk is 2147483648 bytes, the path of the memory disk is set to / mnt / mem cache, and the path of the NVME disk is set to / mnt / nvme_cache.

[0163] Exemplarily, if it is necessary to cache the data of multiple files under a file directory to the client's cache disk, the parameters in the execution statement can be {'subdir':' / whml','file_path':' / tt3','cache_type':'1'}, wherein subdir indicates that the client's directory is / whml, file_path indicates that the file path of the files under the directory on the client is / tt3, and a cache_type value of 1 indicates that the data of multiple files under the directory obtained by the server is cached to the client's memory disk. If the cache_type value is 2, it indicates that the data of multiple files under the directory obtained by the server is cached to the client's NVME disk.

[0164] In a possible implementation, the client may have more than one NVME disk.

[0165] Another data processing method provided by the present application enables the cache management module of the client to pre-add the file data of the file directory information to the cache disk of the client, so that after the processing module receives the first request from the target application, it can directly read from the cache disk of the client, instead of reading from the OST corresponding to the Lustre file system, which conveniently and quickly improves the efficiency of reading massive small file data. In addition, the cache management module of the client manages the cache disk independently of the processing module, providing the processing module with more thread resources for reading massive small file data, thereby improving its reading efficiency.

[0166] Figure 7 A schematic diagram of the structure of the data processing device provided in this application, such as Figure 7 As shown, the data processing device 70 provided by the present application includes: a processing module 701, a cache management module 702;

[0167] The processing module 701 is used to receive a first request from a target application, where the first request is used to request to read data of a first file; the file size of the first file is smaller than a preset value.

[0168] The processing module 701 is further used to determine whether there is a cache file of the first file in the cache disk of the client.

[0169] The processing module 701 is further configured to determine whether the cache file is valid if it is determined that there is a cache file of the first file.

[0170] The processing module 701 is further configured to read the cache file of the first file from the cache disk of the client to obtain data of the first file if it is determined that the cache file of the first file is valid.

[0171] The processing module 701 is further configured to send the data of the first file to the target application.

[0172] In a possible implementation, the processing module 701 is further configured to: search for metadata information of the first file according to the identifier of the first file;

[0173] If the metadata information of the first file includes a cache path of the first file on the cache disk of the client, it is determined that a cache file of the first file exists on the cache disk of the client.

[0174] In a possible implementation, the processing module 701 is further used to: obtain, from the metadata information of the first file, the latest modification time of the first file on the cache disk of the client and the latest modification time of the first file on the server;

[0175] If the latest modification time of the first file on the cache disk of the client is consistent with the latest modification time of the first file on the server, it is determined that the cache file of the first file is valid.

[0176] In a possible implementation, the processing module 701 is further configured to:

[0177] Obtaining a cache path of the first file on a cache disk of the client from metadata information of the first file;

[0178] Open the cache file of the first file in the cache disk of the client according to the cache path;

[0179] A handle of the cache file is obtained, and the cache file of the first file is read from the cache disk of the client according to the handle of the cache file to obtain data of the first file.

[0180] In a possible implementation, the processing module 701 is further used to: determine whether the first request carries a first mark, where the first mark is used to indicate a write operation to be performed on the data of the first file;

[0181] If it is determined that the first request does not carry the first mark, it is determined whether a cache file of the first file exists in the cache disk of the client.

[0182] In a possible implementation, the processing module 701 is further used to: receive a second request from the target application, where the second request is used to request to close the first file;

[0183] Determine whether the cache file of the first file in the cache disk of the client is opened;

[0184] If it is determined that the cache file of the first file in the cache disk of the client is opened, the handle of the cache file is deleted, and the cache file is closed.

[0185] In a possible implementation, the processing module 701 is further configured to: if a preset condition is met, obtain a file path of the first file on the server from metadata information of the first file;

[0186] Obtain data of the first file from the server according to the file path of the first file on the server;

[0187] The preconditions include at least one of the following:

[0188] The cache file of the first file does not exist in the cache disk of the client;

[0189] The cache file of the first file is invalid;

[0190] The first request carries a first tag.

[0191] The cache management module 702 is used to receive a third request from the cache management terminal, the third request including file directory information, the file directory information including an identifier of the file directory and identifiers of multiple files under the file directory; the third request is used to add data of multiple files under the directory to the cache disk of the client in advance;

[0192] According to the file directory information, the data of multiple files in the file directory are obtained from the server, and the data of multiple files are cached to the cache disk of the client.

[0193] The cache management module 702 is further configured to send indication information to the cache management module when the processing module determines that the latest modification time of the first file on the cache disk of the client is inconsistent with the latest modification time of the first file on the server;

[0194] The cache management module performs at least one of the following according to the instruction information:

[0195] Deleting a cache file of the first file in the cache disk of the client from the cache disk of the client;

[0196] The latest data of the first file is obtained from the server, and the latest data of the first file is cached to the cache disk of the client.

[0197] The cache management module 702 is also used to cache file data to the NVME disk when the storage space of the memory disk is insufficient.

[0198] Figure 8 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 8 As shown, the electronic device 80 provided in the embodiment of the present application includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0199] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0200] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0201] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0202] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0203] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0204] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0205] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0206] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0207] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0208] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0209] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0210] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0211] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0212] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0213] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0215] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: Applied to a client, the client includes a processing module and a target application, and the method includes: The processing module receives a first request from the target application, where the first request is used to request to read data of a first file; and the file size of the first file is smaller than a preset value; The processing module determines whether a cache file of the first file exists in the cache disk of the client; If it is determined that there is a cache file of the first file, the processing module determines whether the cache file is valid; If it is determined that the cache file of the first file is valid, the processing module reads the cache file of the first file from the cache disk of the client to obtain data of the first file; The processing module sends the data of the first file to the target application.

2. The method according to claim 1, characterized in that The first request includes an identifier of the first file, and the processing module determines whether a cache file of the first file exists in the cache disk of the client, including: The processing module searches for metadata information of the first file according to the identifier of the first file; If the metadata information of the first file includes a cache path of the first file on the cache disk of the client, it is determined that a cache file of the first file exists on the cache disk of the client.

3. The method according to claim 1 or 2, characterized in that: The processing module determines whether the cache file of the first file is valid, including: The processing module obtains the latest modification time of the first file on the cache disk of the client and the latest modification time of the first file on the server from the metadata information of the first file; If the latest modification time of the first file in the cache disk of the client is consistent with the latest modification time of the first file in the server, the processing module determines that the cache file of the first file is valid.

4. The method according to any one of claims 1 to 3, characterized in that: The processing module reads the cache file of the first file from the cache disk of the client to obtain data of the first file, including: The processing module acquires a cache path of the first file on the cache disk of the client from metadata information of the first file; The processing module opens the cache file of the first file in the cache disk of the client according to the cache path; The processing module obtains the handle of the cache file, and reads the cache file of the first file from the cache disk of the client according to the handle of the cache file to obtain data of the first file.

5. The method according to any one of claims 1 to 4, characterized in that: The processing module determines whether a cache file of the first file exists in the cache disk of the client, including: The processing module determines whether the first request carries a first mark, where the first mark is used to indicate a write operation is to be performed on data of the first file; If it is determined that the first request does not carry the first tag, the processing module determines whether a cache file of the first file exists in the cache disk of the client.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The processing module receives a second request from the target application, where the second request is used to request closing the first file; The processing module determines whether the cache file of the first file in the cache disk of the client has been opened; If it is determined that the first file has been opened in the cache file of the cache disk of the client, the processing module deletes the handle of the cache file and closes the cache file.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: If the preset condition is met, the processing module obtains the file path of the first file on the server from the metadata information of the first file; The processing module obtains data of the first file from the server according to the file path of the first file on the server; The preset condition includes at least one of the following: The cache disk of the client does not have a cache file of the first file; The cache file of the first file is invalid; The first request carries a first tag.

8. The method according to claims 1 to 6, characterized in that The client further includes a cache management module, and the method further includes: The cache management module receives a third request from the cache management terminal, the third request including file directory information, the file directory information including an identifier of the file directory and identifiers of multiple files under the file directory; the third request is used to add data of the multiple files under the directory to the cache disk of the client in advance; The cache management module obtains data of multiple files in the file directory from the server according to the file directory information, and caches the data of the multiple files to the cache disk of the client.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: When the processing module determines that the latest modification time of the first file on the cache disk of the client is inconsistent with the latest modification time of the first file on the server, the processing module sends indication information to the cache management module; The cache management module performs at least one of the following according to the indication information: Deleting a cache file of the first file in the cache disk of the client from the cache disk of the client; The latest data of the first file is obtained from the server, and the latest data of the first file is cached to the cache disk of the client.

10. The method according to any one of claims 1 to 8, characterized in that: The cache disk of the client includes a memory disk and a non-volatile memory standard NVME disk, and the method further includes: When the storage space of the memory disk is insufficient, the cache management module caches the file data to the NVME disk.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

Images