Data processing method and device, storage medium and electronic equipment

Abstract

The invention provides a data processing method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a data processing request which is used for requesting to read or store data corresponding to a target file; obtaining metadata corresponding to the target file from a target database according to the data processing request; and responding to the data processing request according to the metadata. When the data processing request is obtained, by obtaining the metadata in the target database, the time delay in the metadata reading process is reduced, the read-write performance of the target file is improved, and then the data processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, storage medium and electronic device. Background Art

[0002] In order to meet the data storage needs of different organizations, distributed storage technology has emerged. Distributed storage is a technology that connects multiple computers or storage devices through a network to jointly store and manage data. It stores data on multiple nodes, each of which can be a physical machine, a virtual machine, or a cloud storage service. In distributed storage technology, data storage is achieved through a file system. The file system divides the disk space into multiple file blocks, and maps the logical blocks of the file to the physical blocks based on the data management information. The data management information is the metadata in the distributed storage system. When reading and writing data, the metadata should be obtained first to obtain the data storage method, and then the data should be stored according to the metadata.

[0003] Typically, metadata is stored in a separate server. When storing data, it is necessary to read metadata from a remote storage server through remote procedure call technology, which results in a certain delay in the metadata reading process and reduces the efficiency of data storage. Summary of the invention

[0004] In view of this, the present disclosure provides a data processing method, device, storage medium and electronic device, which can improve the efficiency of data storage and data reading.

[0005] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:

[0006] Obtaining a data processing request, wherein the data processing request is used to request reading or storing data corresponding to a target file;

[0007] According to the data processing request, obtaining metadata corresponding to the target file in a target database;

[0008] The data processing request is responded to according to the metadata.

[0009] In one embodiment, obtaining metadata corresponding to the target file in a target database according to the data processing request includes:

[0010] Determining the name of the target file according to the data processing request;

[0011] Performing hash calculation on the name of the target file to obtain an index key corresponding to the target file;

[0012] According to the index key, metadata corresponding to the target file is obtained in the target database.

[0013] In one embodiment, acquiring metadata corresponding to the target file in the target database according to the index key includes:

[0014] According to the index key, searching the memory table of the target database for metadata corresponding to the target file;

[0015] If the metadata corresponding to the target file does not exist in the memory table, searching the metadata corresponding to the target file in the immutable memory table;

[0016] When the metadata corresponding to the target file does not exist in the immutable memory table, the metadata corresponding to the target file is retrieved in the ordered string table.

[0017] In one embodiment, when the data processing request is used to request to store data corresponding to the target file, responding to the data processing request according to the metadata includes:

[0018] Generate a write request corresponding to the target file according to the metadata, wherein the write request is a write request received by an I / O channel in a storage performance development toolkit SPDK environment;

[0019] Write the data corresponding to the target file into the data buffer in the SPDK environment;

[0020] According to the write request, the data in the data buffer is written to the hard disk through an asynchronous processing program in the SPDK environment.

[0021] In one embodiment, when the data processing request is used to request to read data corresponding to the target file, responding to the data processing request according to the metadata includes:

[0022] Generate a read request corresponding to the target file according to the metadata, wherein the read request is a read request received by an I / O channel in an SPDK environment;

[0023] According to the read request, the data in the hard disk is read into the data buffer through an asynchronous processing program;

[0024] Returns the data in the data buffer.

[0025] In one embodiment, the method further comprises:

[0026] Get the data of the target file;

[0027] Performing hash calculation on the file name of the target file to obtain an index key corresponding to the target file;

[0028] Determining metadata corresponding to the target file based on the data of the target file;

[0029] The index key corresponding to the target file and the metadata are stored in the target database accordingly.

[0030] In one embodiment, obtaining data of the target file includes:

[0031] Get multiple initial files;

[0032] The multiple initial files are aggregated to obtain data of the target file.

[0033] In one embodiment, storing the index key corresponding to the target file and the metadata corresponding to the target file in the target database includes:

[0034] The index key and the metadata corresponding to the target file are stored in the memory table accordingly;

[0035] When the amount of data in the memory table reaches a first threshold, converting the memory table into an immutable memory table;

[0036] When the number of the immutable memory tables reaches a second threshold, the immutable memory tables are persisted to the ordered string table.

[0037] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, the device comprising:

[0038] A first acquisition unit, configured to acquire a data processing request, wherein the data processing request is used to request to read or store data corresponding to a target file;

[0039] A second acquisition unit, configured to acquire metadata corresponding to the target file in a target database according to the data processing request;

[0040] A response unit is used to respond to the data processing request according to the metadata.

[0041] In one embodiment, the second acquiring unit is specifically configured to:

[0042] Determining the name of the target file according to the data processing request;

[0043] Performing hash calculation on the name of the target file to obtain an index key corresponding to the target file;

[0044] According to the index key, metadata corresponding to the target file is obtained in the target database.

[0045] In one embodiment, the second acquiring unit is specifically configured to:

[0046] According to the index key, searching the memory table of the target database for metadata corresponding to the target file;

[0047] If the metadata corresponding to the target file does not exist in the memory table, searching the metadata corresponding to the target file in the immutable memory table;

[0048] When the metadata corresponding to the target file does not exist in the immutable memory table, the metadata corresponding to the target file is retrieved in the ordered string table.

[0049] In one embodiment, the response unit is specifically used to:

[0050] Generate a write request corresponding to the target file according to the metadata, wherein the write request is a write request received by an I / O channel in a storage performance development toolkit SPDK environment;

[0051] Writing the data corresponding to the target file into a data buffer, wherein the data buffer refers to a data buffer in the SPDK environment;

[0052] According to the write request, the data in the data buffer is written to the hard disk through an asynchronous processing program in the SPDK environment.

[0053] In one embodiment, the response unit is specifically used to:

[0054] Generate a read request corresponding to the target file according to the metadata, wherein the read request is a read request received by an I / O channel in an SPDK environment;

[0055] According to the read request, the data in the hard disk is read into the data buffer through an asynchronous processing program;

[0056] Returns the data in the data buffer.

[0057] In one embodiment, the apparatus further comprises:

[0058] A third acquisition unit, used for acquiring data of a target file;

[0059] A calculation unit, used to perform a hash calculation on the file name of the target file to obtain an index key corresponding to the target file;

[0060] A determination unit, configured to determine metadata corresponding to the target file based on the data of the target file;

[0061] The storage unit is used to store the index key corresponding to the target file and the metadata in the target database accordingly.

[0062] In one embodiment, the third acquisition unit is specifically configured to:

[0063] Get multiple initial files;

[0064] The multiple initial files are aggregated to obtain data of the target file.

[0065] In one embodiment, the storage unit is specifically used for:

[0066] The index key and the metadata corresponding to the target file are stored in the memory table accordingly;

[0067] When the amount of data in the memory table reaches a first threshold, converting the memory table into an immutable memory table;

[0068] When the number of the immutable memory tables reaches a second threshold, the immutable memory tables are persisted to the ordered string table.

[0069] According to a third aspect of an embodiment of the present disclosure, there is provided a non-temporary computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any method described in the first aspect are implemented.

[0070] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any method described in the first aspect are implemented.

[0071] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0072] When a data processing request is received, metadata is obtained from the target database, which reduces the latency in the metadata reading process, improves the read and write performance of the target file, and thus improves the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a flow chart of a data processing method shown in an exemplary embodiment of the present disclosure;

[0074] Figure 2 is a flow chart of a method for obtaining metadata shown in an exemplary embodiment of the present disclosure;

[0075] Figure 3 is a flow chart of a method for configuring an SPDK environment shown in an exemplary embodiment of the present disclosure;

[0076] Figure 4 is a flow chart of a method for storing a target file shown in an exemplary embodiment of the present disclosure;

[0077] Figure 5 is a flowchart of a method for storing metadata shown in an exemplary embodiment of the present disclosure;

[0078] Figure 6 is a block diagram of a data processing device shown in an exemplary embodiment of the present disclosure;

[0079] Figure 7 is a structural diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0081] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0082] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0083] With the popularization and coverage of the Internet, people's daily lives have begun to be fully connected to the Internet. As a result, the amount of data has increased dramatically and the data types have become more diverse. Big data, cloud computing, and artificial intelligence have also developed rapidly, and more and more organizations need to process and store massive amounts of data. Therefore, distributed storage technology has gradually become the key to managing large-scale data centers. Distributed storage is a technology that connects multiple computers or storage devices through a network to jointly store and manage data. It stores data on multiple nodes, each of which can be a physical machine, a virtual machine, or a cloud storage service.

[0084] In distributed storage technology, the process of data storage and reading is implemented through the file system. The file system reads and writes data based on the disk space layout and data management. The disk space management contains the space usage information and realizes the division of files into multiple file blocks. Data management maps the logical blocks of files to the physical blocks through management information. This management information is metadata. When managing file data, it is necessary to first obtain metadata to obtain information such as the storage method and storage location of the file data.

[0085] In traditional methods, file data is usually stored in mechanical hard disks. Although mechanical hard disks are affordable, their hardware performance is poor. The speed of data transmission from mechanical hard disks to memory is slow and the latency is high. In addition, when reading data, the user program must first fall into the kernel, trigger a system call, and then go through the scheduling of the virtual file system and the kernel file system, and finally obtain the required resources from the local disk device. The entire IO path is long. Therefore, the IO operations in the process of reading and writing metadata will increase the IO latency of processing file data and reduce IO performance. At the same time, there is a certain network transmission delay in the acquisition of metadata, which will also cause a reduction in IO performance.

[0086] Based on this, the present disclosure provides a data processing method, which stores metadata based on RocksDB (key-value storage engine) technology, and calls the metadata in RocksDB through SPDK (Storage Performance Development Kit) technology to achieve storage of file data. This method can be applied to electronic devices, such as computers or central servers, to reduce the delay of reading and writing metadata. Figure 1 According to a flowchart of a data processing method shown in an exemplary embodiment, the flowchart may include the following steps:

[0087] S101, obtaining a data processing request, wherein the data processing request is used to request to read or store data corresponding to a target file.

[0088] In some embodiments, the data management request is a request initiated by the client to request to read or store a target file. For example, an application calls a request for file data or a request to store file data. The target file refers to a data file that needs to be stored or read according to the operation of the client. The target file may be a data file generated by the application, or it may be a data file required for the operation of the application. According to the data management request, information such as the file name and file type of the target file that the client needs to access can be determined. In a distributed storage system, a computer or server for managing data receives a data processing request initiated by a client, and stores or reads the data in the target file according to the data processing request.

[0089] S102: Acquire metadata corresponding to the target file in a target database according to the data processing request.

[0090] In some embodiments, metadata indicating the storage location, file block pointer, file owner, access rights, file type and other information of the file is stored in the target database. The target database can be a database built based on the RocksDB key-value storage engine. RocksDB is an open source, high-performance, persistent key-value storage engine that supports fast read and write operations, has high reliability and scalability, and effectively supports various data-intensive applications. After obtaining the data processing request initiated by the client, the index key of the target file in the target database can be determined according to the file name of the target file in the data processing request, and the metadata corresponding to the target file can be obtained in the target database where the metadata is stored according to the index key.

[0091] S103: respond to the data processing request according to the metadata.

[0092] In the disclosed embodiment, the type of the target file and the storage location of the target file in the solid-state drive can be determined based on the metadata corresponding to the target file. An IO request can be constructed based on the acquired metadata, and the IO request can be submitted to the IO channel of SPDK, so that SPDK asynchronously processes the IO request, reads or writes the data corresponding to the target file, and completes the response to the data processing request. SPDK technology can realize data transmission and storage operations in user mode, avoiding the frequent context switching between the kernel mode and the user mode of the traditional operating system, thereby improving the efficiency of data transmission and storage operations.

[0093] When a data processing request is received, metadata is obtained from the target database, which reduces the latency in the metadata reading process, improves the read and write performance of the target file, and thus improves the efficiency of data processing.

[0094] In an embodiment of the present disclosure, obtaining metadata corresponding to the target file in a target database based on the data processing request includes: determining a name of the target file based on the data processing request; performing a hash calculation on the name of the target file to obtain an index key corresponding to the target file; and obtaining metadata corresponding to the target file in the target database based on the index key.

[0095] In some embodiments, according to the data processing request initiated by the client, the name of the target file can be determined first. Then, the name of the target file is hashed according to the hash function to obtain the hash value corresponding to the target file, and the hash value is determined as the index key corresponding to the target file. The selection of the hash function can be selected according to the file storage requirements. For example, the hash value corresponding to the name of the target file can be quickly calculated by the MD5 or MurmurHash function. By calculating the hash value of the target file through the hash function, the file name can be evenly mapped to the hash value space to ensure that the distribution of each hash value is good and reduce the probability of collision (i.e., multiple different file names are mapped to the same hash value). Because in the target database, metadata and index keys are one-to-one corresponding, the metadata corresponding to the target file can be quickly obtained in the target database according to the index key of the target file.

[0096] Using RocksDB database technology to manage metadata effectively improves the IO efficiency of obtaining metadata. The metadata corresponding to the target file is obtained in the target database according to the hash value calculated from the target file name, which improves the storage and retrieval efficiency of the metadata corresponding to the target file.

[0097] In the disclosed embodiment, obtaining metadata corresponding to the target file in the target database based on the index key includes: searching for metadata corresponding to the target file in a memory table of the target database based on the index key; if the metadata corresponding to the target file does not exist in the memory table, searching for metadata corresponding to the target file in an immutable memory table; if the metadata corresponding to the target file does not exist in the immutable memory table, searching for metadata corresponding to the target file in an ordered string table.

[0098] In some embodiments, after determining the index key corresponding to the target file, the metadata corresponding to the target file is searched in the target database according to the index key. First, the metadata is searched according to the index key in the memory table (Memtable), which is a data structure located in the memory of the database system, used to temporarily store data to be written to the disk, allowing fast insertion, update and deletion operations, and is a key component for achieving fast writing in the database.

[0099] If the metadata corresponding to the target file does not exist in the memory table, the metadata corresponding to the target file will continue to be retrieved from the immutable memtable according to the index key. Immutable memtable is a special memtable that will not be modified once created, nor will it be updated in place, providing a more efficient and secure way to update data. Because there is no need to modify data in place, data competition and lock problems are avoided, and concurrent operations are also facilitated.

[0100] If the metadata corresponding to the target file does not exist in the immutable memory table, the metadata corresponding to the target file is retrieved in the SST (Sorted String Table) file according to the index key. When retrieving the metadata corresponding to the target file in the SST file, you can search layer by layer from low to high according to the hierarchical order of the SST to obtain the metadata corresponding to the target file. The SST file is an ordered key-value pair file, which is usually used to persist data in database systems. These files store data after being written to disk from the memtable. The key-value pairs inside the SST file are sorted in the order of the keys, which can speed up range query operations and facilitate fast data retrieval.

[0101] In one embodiment, the process of obtaining metadata in the target database according to the data processing request can refer to Figure 2 The flowchart shown includes the following steps:

[0102] S201, obtaining the file name of the target file according to the data processing request.

[0103] S202, performing hash calculation according to the file name to obtain a corresponding index key value (key value).

[0104] S203, searching the memory table for records to determine whether corresponding metadata is found.

[0105] S204: If the metadata corresponding to the target file is found, the result is returned.

[0106] S205: If the metadata corresponding to the target file does not exist in the memory table, search for records in the immutable memory table to determine whether the corresponding metadata is found.

[0107] S206, if the metadata corresponding to the target file does not exist in the immutable memory table, the metadata corresponding to the target file is searched in the ordered string table.

[0108] In an embodiment of the present disclosure, when the data processing request is used to request the storage of data corresponding to a target file, responding to the data processing request based on the metadata includes: generating a write request corresponding to the target file based on the metadata; writing the data corresponding to the target file to a data buffer in an SPDK environment; and writing the data in the data buffer to a hard disk through an asynchronous handler in the SPDK environment based on the write request.

[0109] In some embodiments, when the data processing request is for storing data corresponding to a target file, the SPDK technology can be used to store the data in the target file to the solid-state drive according to the metadata corresponding to the target file. Before using the SPDK technology to store the data in the target file, the SPDK application environment needs to be configured. The specific process can be referred to Figure 3 The flowchart shown in FIG. 1 includes the following steps:

[0110] S301, obtaining the current SSD (Solid State Disk or Solid State Drive) drive information, the drive information may include: the capacity, interface type, read and write speed, durability and controller type of the SSD.

[0111] S302, switching the SSD driver to UIO (Userspace I / O), so that the SSD driver runs as a user space instead of being executed in the kernel space.

[0112] S303, initialize the SPDK environment and set the SPDK memory pool, which specifically includes creating an asynchronous I / O processing thread in the SPDK and allocating an I / O channel to each asynchronous I / O processing thread.

[0113] S304, initialize SSD information and register the SSD information in the SPDK environment.

[0114] S305, instantiate the SSD object so that the SSD can interact with the SPDK.

[0115] S306, connect the SSD disk to the SPDK environment.

[0116] Through the above steps, the storage process of the target file can be migrated from the kernel to the user state, and the solid-state drive can transmit data with SPDK, ensuring that the data in the target file can be written to the solid-state drive using SPDK technology.

[0117] After the SPDK environment is set up, the electronic device used to store the target file can generate a write request corresponding to the target file based on the metadata corresponding to the target file. The write request contains the metadata corresponding to the target file and is an I / O request that can be received by the I / O channel. In addition, the data in the target file is written to the data buffer of the SPDK. Then the write request is submitted to the I / O channel of the SPDK, so that the SPDK starts an asynchronous handler to process the write request and writes the data in the data buffer to the hard disk.

[0118] After writing the data in the data buffer to the SSD, mark the data buffer as reusable or reclaimed, reclaim the data buffer, clean up the SSD instance, disconnect the SSD from SPDK, close the SSD registration information in SPDK, and clean up the SPDK environment.

[0119] By storing the target file through SPDK technology, the storage process of the target file can be migrated from kernel state to user state, shortening the IO path in the file storage process and improving the efficiency of file storage.

[0120] In one embodiment, the process of storing the target file through SPDK technology refers to Figure 4 The flowchart shown in the figure has the following specific steps:

[0121] S401, initialize the SPDK environment and configure resources. For details, see Figure 3 Flowchart shown.

[0122] S402: Obtain metadata corresponding to the target file according to the data processing request initiated by the client.

[0123] S403: Generate a write request (I / O request) corresponding to the target file according to the metadata.

[0124] S404, writing the data in the target file into the data buffer managed by SPDK.

[0125] S405: Submit the write request to the IO channel.

[0126] S406, start the SPDK asynchronous processing program to process the write request, and write the data in the data buffer to the solid state drive.

[0127] S407, reclaiming the data buffer.

[0128] S408, clean up the SPDK environment.

[0129] In an embodiment of the present disclosure, when the data processing request is used to request reading data corresponding to a target file, responding to the data processing request based on the metadata includes: generating a read request corresponding to the target file based on the metadata, wherein the read request is a read request received by an I / O channel in an SPDK environment; reading data from a hard disk into a data buffer through an asynchronous processing program based on the read request; and returning the data in the data buffer.

[0130] In some embodiments, when the data processing request is for reading data corresponding to a target file, the data corresponding to the target file stored in the hard disk can be read by using the SPDK technology. It should be noted that before using the SPDK technology to read the data stored in the hard disk, it is also necessary to execute Figure 3 The steps shown are used to configure the application environment of SPDK. After the SPDK environment configuration is completed, the electronic device used to read the target file can generate a read request corresponding to the target file based on the metadata corresponding to the target file. The metadata corresponding to the target file exists in the read request, which is an I / O request that can be received by the I / O channel. By submitting the read request to the I / O channel of SPDK, SPDK starts an asynchronous handler to process the read request, and the data corresponding to the target file stored in the hard disk is read into the data buffer of SPDK. After the reading is completed, the initial data processing request can be responded to, and the data corresponding to the target file stored in the data buffer can be returned to the client that initiated the data processing request.

[0131] By reading the target file through SPDK technology, the target file reading process can be migrated from kernel state to user state, shortening the IO path in the file reading process and improving the efficiency of file reading.

[0132] In the disclosed embodiment, the method further includes: acquiring data of the target file; performing hash calculation on the file name of the target file to obtain an index key corresponding to the target file; determining metadata corresponding to the target file based on the data of the target file; and storing the index key and metadata corresponding to the target file in the target database.

[0133] In some embodiments, before obtaining metadata in the target database, the metadata corresponding to the target file needs to be stored in the target database. The following is an explanation of the process of storing the metadata corresponding to the target file in the target database. First, by obtaining the data of the target file, the file type, file name, file size, file path and other information of the target file can be determined. Then, a hash calculation is performed according to the file name of the target file to obtain a hash value corresponding to the target file, and the hash value is determined as the index key of the target file in the target database. Then, according to the file type, file size, file path and other information of the target file and the preset file storage rules, the metadata corresponding to the target file is determined. The file storage rules refer to the storage rules determined when the functions of each storage node are divided in the distributed storage system. At this point, the index key and the corresponding metadata of the target file in the target database can be determined, and finally the index key and metadata are stored in the target database accordingly, so that the key-value pairs corresponding to the target file in the target database are stored. There are multiple key-value pairs stored in the target database, each of which is the index key and metadata of a file. According to the index key, the metadata of the target file can be quickly retrieved in the target database.

[0134] By calculating the hash value based on the file name of the target file and using the hash value as the index key to store metadata, the corresponding data item can be quickly located without the need for sequential search or complex index structure. In addition, the impact of collisions can be effectively reduced, ensuring the efficiency and stability of data operations and protecting data security and privacy.

[0135] In the disclosed embodiment, the obtaining of the target file includes: obtaining a plurality of initial files; and aggregating the plurality of initial files to obtain data of the target file.

[0136] In some embodiments, the files that need to be stored in the client may be files with a small amount of data (for example, files with a file size less than 64MB). When storing the metadata of these small files, if the metadata of the small files is stored separately one by one, the storage efficiency of the metadata may be affected. Therefore, for small files with the same file type and the same corresponding metadata, after obtaining the initial file of each small file, multiple initial files can be aggregated based on the file type and file size to obtain the target file. When aggregating files, the data volume can be used as a threshold, for example, the size of the aggregated target file is not less than 1GB, or the file data can be used as a threshold, for example, 10 initial files are aggregated. After the aggregation is completed, the multiple initial files are regarded as a target file, and then a hash calculation can be performed according to the file name of the target file to obtain the hash value corresponding to the target file, and the hash value is determined as the index key of the target file in the target database. Then, according to the file type, file size, file path and other information of the target file and the preset file storage rules, the metadata corresponding to the target file is determined. At this point, the index key and the corresponding metadata of the target file in the target database can be determined, and finally the index key and metadata are stored in the target database accordingly, so that the key-value pair corresponding to the target file in the target database is obtained. It should be noted that the key-value pair corresponding to each initial file can be determined based on the key-value pair corresponding to the target file. When reading the metadata, the index value can be calculated based on the file name of each initial file, and the metadata corresponding to each initial file can be obtained in the target database based on the index value.

[0137] By aggregating multiple initial files and storing metadata of the multiple initial files in the aggregated target file, the storage efficiency of the multiple initial files is improved.

[0138] In the embodiment of the present disclosure, the storing the index keys corresponding to the target files and the metadata corresponding to the target database includes: storing the index keys corresponding to the target files and the metadata corresponding to the target files in a memory table; when the amount of data in the memory table reaches a first threshold, converting the memory table into an immutable memory table; when the number of immutable memory tables reaches a second threshold, persisting the immutable memory table to an ordered string table.

[0139] In some embodiments, when storing the index keys and metadata corresponding to the target files in the target database, the key-value pairs representing the index keys and metadata are written into the memory table of the target database, and then it is determined whether the amount of data in the memory table reaches a preset first threshold. If the amount of data in the memory table reaches the first threshold, the data table is converted into an immutable memory table, and it is further determined whether the amount of data in the immutable memory table in the target database reaches a second threshold. If the number of immutable memory tables in the target database reaches the second threshold, the data in the immutable memory table is persisted in an ordered string table, thereby achieving the storage of the key-value pairs corresponding to the target files in the target database.

[0140] In one embodiment, the metadata storage process can refer to Figure 5 The flowchart shown in the figure has the following specific steps:

[0141] S501, obtaining multiple small files.

[0142] S502, aggregating multiple small files according to information such as file type and file size to obtain a target file.

[0143] S503: Determine metadata corresponding to the target file according to data information such as the file type and path information of the target file and a preset file storage rule.

[0144] S504, performing a hash calculation according to the file name of the target file to obtain an index key corresponding to the target file.

[0145] S505, writing the key-value pair carrying the index key and metadata information of the target file into the WAL (Write-Ahead Logging) file of the RocksDB database.

[0146] S506: Write the key-value pair carrying the index key and metadata information of the target file into the memory table of the RocksDB database.

[0147] S507, determining whether the amount of data in the memory table reaches a threshold.

[0148] S508: If the amount of data in the memory table reaches a threshold, the memory table is converted into an immutable memory table, and a new memory table is generated for subsequent key-value pair writing.

[0149] S509: Determine whether the number of immutable memory tables in the RocksDB database reaches a threshold.

[0150] S510, if the number of immutable memory tables in the RocksDB database reaches a threshold, scan the key-value pairs in all the immutable memory tables and persist the key-value pairs in the immutable memory tables into data blocks.

[0151] S511, determining whether the file size of the data block reaches a preset threshold, or whether the key-value pairs are completely stored.

[0152] S512: If yes, a compression algorithm is used to compress the data in the data block, and a persistence operation of the data block is triggered.

[0153] S513, write the key-value pairs in the data block into the ordered string table to complete the storage of the metadata.

[0154] Corresponding to the embodiments of the aforementioned data processing method, the present disclosure also provides embodiments of a data processing device.

[0155] Please refer to Figure 6 , Figure 6 is a block diagram of a data processing device shown in an exemplary embodiment of the present disclosure, which may include the following units:

[0156] A first acquisition unit, configured to acquire a data processing request, wherein the data processing request is used to request to read or store data corresponding to a target file;

[0157] A second acquisition unit, configured to acquire metadata corresponding to the target file in a target database according to the data processing request;

[0158] A response unit is used to respond to the data processing request according to the metadata.

[0159] In the embodiment of the present disclosure, the second acquisition unit is specifically used to:

[0160] Determining the name of the target file according to the data processing request;

[0161] Performing hash calculation on the name of the target file to obtain an index key corresponding to the target file;

[0162] According to the index key, metadata corresponding to the target file is obtained in the target database.

[0163] In the embodiment of the present disclosure, the second acquisition unit is specifically used to:

[0164] According to the index key, searching the memory table of the target database for metadata corresponding to the target file;

[0165] If the metadata corresponding to the target file does not exist in the memory table, searching the metadata corresponding to the target file in the immutable memory table;

[0166] When the metadata corresponding to the target file does not exist in the immutable memory table, the metadata corresponding to the target file is retrieved in the ordered string table.

[0167] In the embodiment of the present disclosure, the response unit is specifically used to:

[0168] Generate a write request corresponding to the target file according to the metadata;

[0169] Write the data corresponding to the target file into the data buffer in the SPDK environment;

[0170] According to the write request, the data in the data buffer is written to the hard disk through an asynchronous processing program in the SPDK environment.

[0171] In the embodiment of the present disclosure, the response unit is specifically used to:

[0172] Generate a read request corresponding to the target file according to the metadata, wherein the read request is a read request received by an I / O channel in an SPDK environment;

[0173] According to the read request, the data in the hard disk is read into the data buffer through an asynchronous processing program;

[0174] Returns the data in the data buffer.

[0175] In the embodiment of the present disclosure, the device further includes:

[0176] A third acquisition unit, used for acquiring data of a target file;

[0177] A calculation unit, used to perform a hash calculation on the file name of the target file to obtain an index key corresponding to the target file;

[0178] A determination unit, configured to determine metadata corresponding to the target file based on the data of the target file;

[0179] The storage unit is used to store the index key corresponding to the target file and the metadata in the target database accordingly.

[0180] In the embodiment of the present disclosure, the third acquisition unit is specifically used to:

[0181] Get multiple initial files;

[0182] The multiple initial files are aggregated to obtain data of the target file.

[0183] In the embodiment of the present disclosure, the storage unit is specifically used for:

[0184] The index key and the metadata corresponding to the target file are stored in the memory table accordingly;

[0185] When the amount of data in the memory table reaches a first threshold, converting the memory table into an immutable memory table;

[0186] When the number of the immutable memory tables reaches a second threshold, the immutable memory tables are persisted to the ordered string table.

[0187] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0188] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0189] The embodiments of the data processing device disclosed in the present invention can be applied to electronic devices. The device embodiments can be implemented by software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of the electronic device in which it is located reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 7 The figure is a hardware structure diagram of an electronic device where the data processing device of the present disclosure is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0190] Based on the same inventive concept, the present disclosure also provides a computer-readable storage medium storing executable instructions, which implement the above-mentioned data processing method when executed by a processor.

[0191] Based on the same inventive concept, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the program.

[0192] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0193] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data processing request, wherein the data processing request is used to request reading or storing data corresponding to a target file; According to the data processing request, obtaining metadata corresponding to the target file in a target database; The data processing request is responded to according to the metadata.

2. The method according to claim 1, characterized in that The step of acquiring metadata corresponding to the target file in a target database according to the data processing request includes: Determining the name of the target file according to the data processing request; Performing hash calculation on the name of the target file to obtain an index key corresponding to the target file; According to the index key, metadata corresponding to the target file is obtained in the target database.

3. The method according to claim 2, characterized in that Acquiring metadata corresponding to the target file in the target database according to the index key includes: According to the index key, searching the memory table of the target database for metadata corresponding to the target file; If the metadata corresponding to the target file does not exist in the memory table, searching the metadata corresponding to the target file in the immutable memory table; When the metadata corresponding to the target file does not exist in the immutable memory table, the metadata corresponding to the target file is retrieved in the ordered string table.

4. The method according to claim 1, characterized in that When the data processing request is used to request to store data corresponding to the target file, responding to the data processing request according to the metadata includes: Generate a write request corresponding to the target file according to the metadata, wherein the write request is a write request received by an I / O channel in a storage performance development toolkit SPDK environment; Write the data corresponding to the target file into the data buffer in the SPDK environment; According to the write request, the data in the data buffer is written to the hard disk through an asynchronous processing program in the SPDK environment.

5. The method according to claim 1, characterized in that When the data processing request is used to request to read data corresponding to the target file, responding to the data processing request according to the metadata includes: Generate a read request corresponding to the target file according to the metadata, wherein the read request is a read request received by an I / O channel in an SPDK environment; According to the read request, the data in the hard disk is read into the data buffer through an asynchronous processing program; Returns the data in the data buffer.

6. The method according to claim 1, characterized in that The method further comprises: Get the data of the target file; Performing hash calculation on the file name of the target file to obtain an index key corresponding to the target file; Determining metadata corresponding to the target file based on the data of the target file; The index key corresponding to the target file and the metadata are stored in the target database accordingly.

7. The method according to claim 6, characterized in that The step of obtaining the data of the target file includes: Get multiple initial files; The multiple initial files are aggregated to obtain data of the target file.

8. The method according to claim 6, characterized in that The storing the index key corresponding to the target file and the metadata corresponding to the target file in the target database includes: The index key corresponding to the target file and the metadata are stored in a memory table accordingly; When the amount of data in the memory table reaches a first threshold, converting the memory table into an immutable memory table; When the number of the immutable memory tables reaches a second threshold, the immutable memory tables are persisted to the ordered string table.

9. A data processing device, characterized in that: The device comprises: A first acquisition unit, configured to acquire a data processing request, wherein the data processing request is used to request to read or store data corresponding to a target file; A second acquisition unit, configured to acquire metadata corresponding to the target file in a target database according to the data processing request; A response unit is used to respond to the data processing request according to the metadata.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images