Data processing method, device, storage medium and electronic equipment

Abstract

The present specification discloses a data processing method, device, storage medium, and electronic device. The method includes: for a stored current data block, if the current data block is input from a file end, saving data block query information for querying the current data block in a data block information list in a first segmentation file, and recording the segmentation file identifier of the first segmentation file in an index node corresponding to a target file to which the first segmentation file belongs; if the current data block is obtained by object conversion input from an object end, saving data block query information for querying the current data block in the object metadata in a data block information list in a second segmentation file based on the object metadata of the object, and recording the segmentation file identifier of the second segmentation file in an index node corresponding to the object.

Description

Technical Field

[0001] This specification relates to the field of data storage, and in particular to a data processing method, device, storage medium, and electronic device. Background Art

[0002] With the rapid development of the internet and the explosive growth of information data in recent years, attention to data storage has been increasing. Traditional centralized file system storage solutions are gradually being replaced by distributed storage systems, including the Hadoop Distributed File System (HDFS), which offers high fault tolerance, low-cost deployment, and high-throughput access to application data; the Google File System (GFS), which utilizes a centralized server model to increase storage capacity while ensuring metadata consistency through master-slave node partitioning; and the General Parallel File System (GPFS), which utilizes cross-node data replication to improve recovery speed in the event of single-point failures in large-scale storage. Distributed storage solutions, represented by file systems, have become a major player in enterprise big data storage and analysis. However, with the widespread application of artificial intelligence technologies and the proliferation of various smart devices, massive amounts of unstructured data with varying sizes and high read / write loads are being generated and processed. File systems, due to their high expansion costs, limited capacity, and inconvenience in sharing, are unable to meet the growing market demand for unstructured data storage.

[0003] Amazon Simple Storage Service (Amazon S3), an object storage solution launched by Amazon, aims to address the aforementioned issues. It manages metadata and data blocks through "objects," flattening the storage structure by indexing objects with globally unique identifiers, thereby simplifying data management. In Amazon S3, users can store and retrieve any amount of object data using a series of HTTP RESTful interfaces provided by the S3 protocol. The resulting object metadata and data blocks are distributed across different disks on distributed nodes according to different redundancy strategies. During expansion, cluster capacity can be rapidly expanded through replication (replica strategy) or EC encoding (erasure coding strategy), depending on the redundancy strategy of the hardware devices housing the data (metadata). Furthermore, the introduction of bucket hashing (bucket sharding) technology enables the object storage solution to theoretically achieve unlimited expansion of the capacity of a single storage bucket, breaking through read and write performance bottlenecks.

[0004] Compared to distributed file systems, distributed object storage offers greater scalability, lower costs, and larger storage capacities, making it a natural fit for storing unstructured data. However, due to its flat storage structure and unique objects, the performance of modifying data after it's written is inevitably poor, making it incapable of handling highly time-sensitive business. To address this, achieving interoperability between object storage and file systems through gateway conversions like S3FS has become a mainstream technology in existing solutions. This solution uses FUSE (Filesystem in Userspace) technology to translate file write operations into object storage upload requests, thereby achieving data interoperability. However, due to the fundamental differences in their storage structures, any data conversion operation will significantly impact read and write performance, while the increase in upload links also introduces the risk of data inconsistency. Therefore, constructing a data format compatible with object storage and file systems is key to solving these problems. Summary of the Invention

[0005] This specification provides a data processing method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a data processing method, which includes:

[0008] For the current data block stored,

[0009] If the current data block is input from the file end, data block query information for querying the current data block is saved in a data block information list in the first segment file, and a segment file identifier of the first segment file is recorded in an index node corresponding to a target file to which the first segment file belongs; the first segment file is recorded in an index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data;

[0010] If the current data block is obtained by converting the object input from the object end, the data block query information for querying the current data block in the object metadata is saved in the data block information list in the second segmentation file according to the object metadata of the object, and the segmentation file identifier of the second segmentation file is recorded in the index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0011] Optionally, the method further includes:

[0012] If the current data block is obtained by converting the object input from the object side and is appended to the file data block uploaded from the file side, a third segmented file is created, data block query information for querying the file data block is saved in a data block information list in the third segmented file, and a segmented file identifier of the third segmented file is recorded in the index node corresponding to the object; the third segmented file is recorded in the index disk; the file data block is one of the data blocks into which the appended data is divided.

[0013] Optionally, the method further includes:

[0014] If the current data block is passed in by the file side and is appended to the object uploaded by the object side, and if the capacity of the last segment file of the target file is less than the set capacity, data block query information for querying each data block in the appended object in the object metadata of the appended object is saved in the data block information list of the last segment file based on the object metadata of the appended object; the appended object is the object corresponding to the appended data uploaded by the object side.

[0015] Optionally, the method further includes:

[0016] If the current data block is transferred from the file end, the object metadata of the target file is created and recorded in the index disk; the object metadata of the target file stores the index node identifier of the index node corresponding to the target file.

[0017] Optionally, the method further includes:

[0018] Determine each index node through the created scanner and pass the index node into the aggregation queue;

[0019] For each index node in the aggregation queue, the index node is received by the created aggregator in the order in which the index nodes are passed into the aggregation queue, and the aggregation flag position of the index node is set to a first flag, where the first flag is used to indicate that the data corresponding to the index node is undergoing aggregation processing;

[0020] Read each segment file in the index node;

[0021] For each segmented file in the index node, the data corresponding to the segmented file is read from the data disk into a preset cache according to the data block information list of the segmented file. When the data stored in the preset cache reaches a preset size, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified; the data to be aggregated is the data stored in the preset cache;

[0022] When aggregation processing is completed for each segment file in the index node, the aggregation flag position of the index node is set to a second flag, and the second flag is used to indicate that the data corresponding to the index node has completed aggregation processing.

[0023] Optionally, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified, specifically including:

[0024] Determine the md5 value of the data to be aggregated;

[0025] The data to be aggregated is stored in the data disk through the aggregator, and the data to be aggregated stored in the data disk is verified according to the md5 value. If the verification passes, the data block information list of the segmented file is modified.

[0026] Optionally, the method further includes:

[0027] For each index node, when the file end reads the data corresponding to the index node for the first time, a pre-read cache instance corresponding to the index node is created;

[0028] Querying a preset number of segment files in the index node from the index disk to obtain segment file identifiers of the segment files and a data block information list of the segment files;

[0029] Filling the segment file identifier of each segment file and the data block information list of each segment file into the pre-read cache instance corresponding to the index node;

[0030] According to the pre-read cache instance corresponding to the index node, the data corresponding to the index node is read.

[0031] This specification provides a data processing device, which includes:

[0032] The file storage module is configured to, for a current data block being stored, save data block query information for querying the current data block into a data block information list in a first segmented file if the current data block is input from a file end, and record a segmented file identifier of the first segmented file into an index node corresponding to a target file to which the first segmented file belongs; the first segmented file is recorded in the index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data;

[0033] The object storage module is used to save, for the current data block stored, data block query information for querying the current data block in the object metadata according to the object metadata of the object, into a data block information list in a second segmentation file, and record the segmentation file identifier of the second segmentation file into an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0034] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0035] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the program.

[0036] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0037] In the data processing method provided in this specification, for the stored current data block, if the current data block is passed in from the file side, the data block query information used to query the current data block is saved in the data block information list in the first segmentation file, and the segmentation file identifier of the first segmentation file is recorded in the index node corresponding to the target file to which the first segmentation file belongs; if the current data block is obtained by object conversion passed in from the object side, the data block query information used to query the current data block in the object metadata is saved in the data block information list in the second segmentation file according to the object metadata of the object, and the segmentation file identifier of the second segmentation file is recorded in the index node corresponding to the object.

[0038] It can be seen from the above method that in the data processing method provided in this specification, the data uploaded by the file side can be converted into each other with the data uploaded by the object side, thereby solving the data compatibility and interoperability problems in cases where the data uploaded by the file side is downloaded by the object side, the data uploaded by the object side is downloaded by the file side, the data uploaded by the file side is appended by the object side, and the data uploaded by the object side is modified by the file side. This essentially solves the differences between the two storage structures and greatly improves the read and write performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0040] Figure 1 A flowchart of a data processing method provided in this specification;

[0041] Figure 2 A schematic diagram of the three forms of metadata of a file data provided in this specification;

[0042] Figure 3 This is a flowchart of a background data aggregation mechanism provided in this manual;

[0043] Figure 4 A flowchart of a pre-read cache mechanism provided in this manual;

[0044] Figure 5 A schematic diagram of a data processing device provided in this specification;

[0045] Figure 6 This manual provides a corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0047] In the prior art, regarding the interoperability between S3 object storage and file system data, the existing gateway conversion technology, S3FS, can write, read, rename, list, and delete object data using file logic. However, due to the fundamental differences between the data structures of "objects" and "files," S3FS suffers significant performance losses when translating data operation requests. Furthermore, the added system complexity reduces data security and increases maintenance costs, making it unsuitable for stable and reliable production applications. To address this issue, the present invention designs a file system metadata format compatible with existing object storage, essentially eliminating the significant structural differences between the two. This allows the object storage system to directly parse and process operation requests from the network file system, thereby simultaneously improving both compatible data read and write performance and data security.

[0048] Based on this, the present specification provides a data processing method, for the current data block stored, if the current data block is passed in by the file side, the data block query information used to query the current data block is saved in the data block information list in the first segmentation file, and the segmentation file identifier of the first segmentation file is recorded in the index node corresponding to the target file to which the first segmentation file belongs; if the current data block is obtained by object conversion passed in by the object side, according to the object metadata of the object, the data block query information used to query the current data block in the object metadata is saved in the data block information list in the second segmentation file, and the segmentation file identifier of the second segmentation file is recorded in the index node corresponding to the object.

[0049] This approach allows data to be processed in a more reliable and performant manner, and both data processed by the file side and the object side can be downloaded by both sides.

[0050] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0051] Figure 1 A flowchart of a data processing method provided in this specification includes the following steps:

[0052] S101: For the current data block stored, if the current data block is passed in from the file end, data block query information used to query the current data block is saved in a data block information list in a first segmented file, and a segmented file identifier of the first segmented file is recorded in an index node corresponding to a target file to which the first segmented file belongs.

[0053] In this specification, the execution entity used to implement a data processing method can be a terminal device such as a laptop computer, a tablet computer, or of course, a server. For the sake of ease of description, this specification only takes the server as the execution entity as an example to illustrate a data processing method provided in this specification.

[0054] For the current data block stored, if the current data block is transferred from the file end, the server saves the data block query information used to query the current data block into the data block information list in the first segment file, and records the first segment file in the index disk. The data block query information used to query the current data block may include the index name of the disk where the data is located, the data block size, and the starting position of the data block reading.

[0055] For example, when storing the first segmented file, it can be stored in the RocksDB database of the index disk in the form of a key-value pair, where the key name (key) is a string composed of the symbol ")", the disk virtual node number (vnode), the bucket name (bucket), and the chunkKey (a string generated by the first segmented file), and the value (value) is the content of the first segmented file itself.

[0056] It should be noted that when storing the target data corresponding to the target file, the data block information list of the first segmented file can record data block query information for multiple data blocks. When the capacity of the first segmented file is greater than or equal to the set capacity, a new segmented file needs to be created and the remaining target data needs to be uploaded to the new segmented file. The capacity of the first segmented file here can refer to the data capacity or the number of data blocks of the first segmented file. For example, if the data capacity of the first segmented file reaches 5GB or the number of data blocks reaches 1024, a new segmented file is created.

[0057] Furthermore, during the upload process, the first segmented file is not recorded to the index disk until its data block information list reaches the set capacity. Instead, the first segmented file is recorded in real time to ensure that no uploaded data blocks are missed, thereby ensuring data reliability. The set capacity of the segmented file is set so that when reading files, the read individual segmented files will not be too large to exceed the size of a single RSocket data packet (RSocket is a network communication technology that can be used for data transmission between distributed nodes).

[0058] Then, the server records the segment file identifier of the first segment file into the index node corresponding to the target file to which the first segment file belongs, wherein the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data.

[0059] For example, when storing an index node, it can be stored in the RocksDB database of the index disk in the form of a key-value pair. The key name (key) is a string composed of the symbol "(", the disk virtual node number (vnode), the bucket name (bucket), and the file index number (nodeId), and the value (value) is the content of the index node itself.

[0060] That is to say, when the server stores the target data corresponding to the target file, a flush is completed every time the data in the file write cache (WriteCache) accumulates to a preset size (e.g., 16MB). The file write cache (WriteCache) is used to temporarily store the uploaded file-side data blocks and achieve write acceleration.

[0061] If there is no data in the index node corresponding to the target file at this time, or the data blocks stored in the last segment file recorded in the index node have reached the capacity limit (e.g., 5G), or the number of data blocks stored in the last segment file has reached the limit (e.g., 1024), a new segment file is generated, the data block query information of the currently stored data block is saved to the data block information list of the new segment file, and the new segment file is recorded in the index disk. Then, the server records the segment file identifier of the new segment file in the index node corresponding to the target file.

[0062] If the last segment file recorded in the index node of the target file has not reached the capacity or quantity limit at this time, the data block query information of the currently stored data block is saved to the data block list of the last segment file, and the record of the last segment file in the index disk is updated.

[0063] Furthermore, if the current data block is transferred from the file client, the server can also create object metadata for the target file and record it in the index disk. For example, if the object metadata is stored, it can be stored in the RocksDB database of the index disk in the form of a key-value pair, where the key name (key) is a string composed of the symbol "*", the disk virtual node number (vnode), the bucket name (bucket), and the object name (objName), and the value (value) is the content of the object metadata itself. The object metadata of the target file stores the index node identifier of the index node corresponding to the target file, which is the file index number (nodeId) mentioned in the above process.

[0064] Based on this, if the current data block is passed in from the file side and is appended to the object uploaded by the object side, if the capacity of the last segment file of the target file is less than the set capacity, the server saves the data block query information used to query each data block in the object metadata of the appended object to the data block information list of the last segment file based on the object metadata of the appended object; if the capacity of the last segment file of the target file is greater than or equal to the set capacity, the server creates a new segment file, saves the data block query information used to query each data block in the object metadata of the appended object to the data block information list of the new segment file, and updates the index node corresponding to the target file (that is, records the segment file identifier of the new segment file to the index node of the target file). The appended object is the object corresponding to the appended data uploaded by the object side.

[0065] S102: For the stored current data block, if the current data block is obtained by converting the object input from the object side, based on the object metadata of the object, save the data block query information for querying the current data block in the object metadata into a data block information list in the second segmentation file, and record the segmentation file identifier of the second segmentation file into the index node corresponding to the object.

[0066] If the current data block is converted from an object input from the object side, that is, if an object input from the object side needs to be operated on by the file side, the server can convert the object input from the object side into a format stored by the file side. Specifically, based on the object metadata of the object, the server saves data block query information for querying the current data block in the object metadata into a data block information list in the second segment file, and records the segment file identifier of the second segment file in the index node corresponding to the object, where the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0067] It should be noted that the object metadata of an object can store data block query information of multiple data blocks. When converting the data corresponding to the object into the storage format of the file end, each data block in the object metadata is traversed, and the data block query information corresponding to each data block is recorded in the second segment file, and the second segment file is recorded in the index disk. At this time, the index node identifier (i.e., nodeId) stored in the object metadata is queried to see if it has a value. If it has a value (i.e., nodeId is greater than 0), the segment file identifier of the second segment file is recorded in the index node corresponding to the object; if it has no value (i.e., nodeId = 0), it can be considered that the object does not exist in the storage format of the file end, and a new index node is generated as the index node corresponding to the object, and the segment file identifier of the second segment file is recorded in the new index node. At the same time, the data block query information recorded in the object metadata of the object needs to be deleted, and the index node identifier of the new index node needs to be recorded in the object metadata.

[0068] In addition, while recording the data block query information corresponding to each data block in the second segment file, the second segment file is recorded in the index disk. When the capacity of the second segment file is greater than or equal to the set capacity (that is, the capacity of the second segment file can refer to the data capacity or data quantity of the second segment file, such as when the data capacity of the second segment file reaches 5G or the number of data blocks reaches 1024), a new segment file needs to be created, and the remaining data blocks in the object continue to be recorded in the new segment file.

[0069] Furthermore, if the current data block is obtained by converting the object input from the object side and is appended to the file data block uploaded from the file side, a third segment file is created, the data block query information used to query the file data block is saved in the data block information list in the third segment file, and the third segment file is recorded in the index disk. The server then records the segment file identifier of the third segment file in the index node corresponding to the object. The file data block is one of the data blocks into which the appended data is divided.

[0070] The first segment file, the second segment file, and the third segment file mentioned in the above process are all metadata used to store part of the data of a large file, which may include the file index number (nodeId), bucket name (bucket), file name (objName), version number (versionNum), the size of the recorded data block (size), and a data block information list used to record the query information of the included data blocks.

[0071] It should be noted that for an object, there must be object metadata corresponding to the object. If the object has not been operated on the file side, there will be no index node corresponding to the object, and the data block query information of the object will be stored in the object metadata. If the object has been operated on the file side, an index node corresponding to the object will be generated, and the data in the object metadata will be transcribed into the index node.

[0072] When a file is initially created, its corresponding object metadata and index nodes are generated simultaneously. Uploading data to the file generates corresponding segment files. When the file is read by the object client, the data blocks corresponding to the index nodes are converted to data blocks corresponding to the object. This allows the object client to read the file. However, the index nodes and segment files recorded on the index disk are not changed; only the data blocks corresponding to the index nodes are temporarily converted to data blocks corresponding to the object.

[0073] It should also be noted that for the same Inode, if a hard link is created, each hard link corresponds to separate object metadata, and all hard links share the same index node identifier (i.e., nodeId), but have different unique enumeration identification numbers (cookies) for easy identification during enumeration.

[0074] For file data stored in the form of index nodes, when reading data from the file end, the server usually needs to query the index disk for each segmented file recorded in the index node based on the segmented file identifier recorded in the index node, and then read the data based on the data block information list of each segmented file. However, if the file data is large, there may be multiple segmented files, and the data block information list of each segmented file records a large number of data blocks. When reading the file data, it is necessary to repeatedly obtain the data block information list of the segmented file from the index disk, which greatly reduces the reading and writing performance. Based on this, the present invention also provides an aggregation processing method, which re-reads the stored file according to a preset merging strategy, merges small data blocks, and then writes them to the disk, thereby achieving the purpose of reducing the number of times the data block query information is read and improving the download performance.

[0075] Specifically, a scanner is created to obtain the index nodes prefixed with "(" in the index disk. Since the same index node is stored in each node in a multi-node storage system, in order to aggregate the current index node in only one node, the server can determine the result of the index node identifier (i.e., the value of nodeId) after taking the modulus. When the result is equal to the file of the current node index number, the data of the current index node is merged. For example, in a three-node storage system, the node index numbers are N = 0, 1, 2, then if the result of nodeId%3 is equal When the index number of the current node is (e.g., nodeId=7, nodeId%3=1, the data corresponding to the index node is aggregated only in the first node), and there is a segmented file with more than 200 data blocks in the index node, the index node is passed to the aggregation queue, where the capacity of the aggregation queue is fixed (e.g., it can accommodate 100 index nodes). If the capacity of the aggregation queue is full, the current scanning position is recorded and the scan is exited. Wait until all index nodes in the aggregation queue complete 50% of the data block aggregation and then wake up the scanner to continue scanning from the recorded scanning position.

[0076] For each index node in the aggregation queue, the created aggregator receives the index node in the order in which the index nodes were entered into the aggregation queue. The aggregation flag for the index node is set to the first mark, where the first mark is used to indicate that the data corresponding to the index node is undergoing aggregation processing. If new data is written at this time, a merge rollback processing request is sent to the node performing the aggregation before the data is written, causing it to immediately stop aggregating the data of the current file, asynchronously delete the aggregated data blocks, and throw the currently processed index node back to the end of the aggregation queue to await the next processing, thereby ensuring the data consistency of the file being written.

[0077] Then, each segmented file in the index node is read. For each segmented file in the index node, the data corresponding to the segmented file is read from the data disk into the preset cache according to the data block information list of the segmented file. When the data stored in the preset cache reaches the preset size (such as 128MB), the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified, where the data to be aggregated is the data stored in the preset cache.

[0078] Specifically, for the index nodes to be aggregated received by the aggregator, each segmented file will be read from the Inode for aggregation according to the currently configured aggregation strategy and the determined concurrency. For example, there may be two aggregation strategies. One is the business priority strategy: this strategy ensures that the foreground business performance is maintained at more than 90% of the background aggregation that is not turned on. At this time, data block aggregation is performed with a concurrency of 1 (the number of segmented files merged at the same time). A single segmented file occupies up to 128MB of cache. After the aggregation is completed, the number of data blocks in the segmented file can be reduced to 1 / 8 of the original; the other is the aggregation priority strategy: this strategy is suitable for execution when the foreground traffic is small, ensuring that the number of small data blocks recorded in the segmented file is minimized. At this time, data block aggregation is performed with a concurrency of 16. A single segmented file occupies up to 128MB of cache. After the aggregation is completed, the number of data blocks in the segmented file can be reduced to 1 / 8 of the original.

[0079] During aggregation, the md5 value of the data block to be aggregated is first calculated for subsequent comparison, and then the corresponding data blocks in the segmented file are read from the data disk into the preset cache in sequence. When the data stored in the preset cache reaches the preset size (such as 128MB), a flush is performed so that the data corresponding to the segmented file can be written to the data disk with the preset size, and the re-writing status of the segmented file is recorded in the index disk. At this time, the new segmented file is recorded in the index disk with a "<" prefix, which is only one prefix different from the original segmented file (the original segmented file is recorded in the index disk with a ")" prefix).

[0080] After the segmented file is aggregated, it is necessary to calculate whether the md5 value of the newly aggregated data block is equal to the original md5 value (for example, the original 8 data blocks are aggregated into 1 data block, whether the md5 value of this 1 data block is equal to the md5 value corresponding to the original 8 data blocks). If there is a difference, the aggregated data block is deleted and thrown back into the aggregation queue to wait for the next execution; if there is no difference, the record of the original segmented file is deleted in the form of a transaction, and the "<" record in the new segmented file is changed to a ")" record, thereby completing the replacement of the segmented file. In addition, if the data corresponding to the read-in segmented file exceeds 128MB during data merging, the last data block will be placed in the next aggregation block for execution.

[0081] Furthermore, when aggregation processing is complete for each segmented file in the index node, the aggregation flag of the index node is set to the second flag, where the second flag is used to indicate that the data corresponding to the index node has completed aggregation processing. To avoid data inconsistency issues, if the system encounters an abnormality such as a node or disk being offline, the current aggregation operation is immediately terminated and the environment is restored. Aggregation will not be resumed until the data and metadata of the files to be aggregated are consistent and there are no abnormalities on each node.

[0082] The background data aggregation mechanism can effectively reduce the number of data blocks recorded in the segmented file, but it cannot reduce the number of times the segmented file is read from the index node. When the number of segmented files in the index node is large, frequent query operations cannot be avoided when downloading the file. In response to this, the present invention also provides a pre-reading method to improve read and write performance.

[0083] Specifically, for each index node, when the file client first reads the data corresponding to that index node, a pre-read cache instance corresponding to that index node is created. The index node identifier of each index node and the pre-read cache instance corresponding to that index node are stored in a common Map collection as key-value pairs, where the key is the nodeId and the value is the pre-read cache instance corresponding to the nodeId. The server then queries the index disk for a preset number of segment files in that index node to obtain the segment file identifiers and data block information lists for each segment file. The preset number here can be 100.

[0084] Then, the server fills the segment file identifier of each segment file and the data block information list of each segment file into the pre-read cache instance corresponding to the index node, and reads the data corresponding to the index node according to the pre-read cache instance corresponding to the index node.

[0085] That is to say, when reading the segmented file in the index node for the first time, the data block information lists corresponding to the preset number of segmented file identifiers will be queried from the index disk at one time, and the segmented file identifiers and the data block information lists corresponding to the segmented file identifiers will be filled into the pre-read cache instance. If the data block information list of the segmented file corresponding to the data to be read does not exist in the cache, the data block information lists corresponding to the preset number of segmented file identifiers will be queried from the index disk again and filled until the data to be read is read. After completing the cache filling operation, the data block information list of the segmented file corresponding to the data to be read this time is taken from the cache to read the data block. After that, each time a read operation is received, the data block information list of the segmented file corresponding to the data to be read will be obtained from the cache first, without querying from the index disk. Only when the segmented file does not exist in the cache will it be queried again from the index disk.

[0086] In addition, when data is written, the cache may become invalid. At this time, the server needs to query the pre-read cache to see whether there is a pre-read cache instance corresponding to the index node currently writing data. If so, the pre-read cache instance is deleted in time so that it can read the new segment file from the index disk again to ensure data consistency.

[0087] With respect to the method in the above description, a complete embodiment is provided here for detailed description.

[0088] First, in this embodiment, there are some noun terms, which are explained here.

[0089] Bucket: The basic entity stored in object storage, a container for storing objects.

[0090] Object: The basic entity stored in object storage. An object is a collection of file data and its related attribute information, including the object key, object data, and object metadata.

[0091] Object metadata (MetaData): Object attribute information stored in the index disk, including information such as object name, object creation date, version number, object data size, object bucket name, object account, etc. This data is used to identify the object and record the storage location of the object data through the data block name (FileName or PartInfo array). In the Hongshan MOSS object storage system, MetaData is stored in the RocksDB database of the index disk in the form of key-value pairs. The key name (key) is a string composed of the symbol "*", the disk virtual node number (vnode), the bucket name (bucket), and the object name (objName). The value (value) is the content of the MetaData itself.

[0092] Disk virtual node (vnode): Any value in the range [0,65535]. It is generated after the disk and the data pool to which the disk belongs are initialized. Each disk has multiple virtual nodes, and a virtual node can only map one disk. The use of virtual nodes and consistent hashing technology can reduce the amount of migrated data when the node device is abnormally disconnected or expanded.

[0093] Inode: A data structure used to store file metadata in a file system. In Macrosan's object storage-compatible file data format, this structure includes attributes such as the file's name (name), size (size), unique index number (nodeId), permissions (mode), user ID (uid), group ID (gid), time (atime / mtime / ctime / atimensec / mtimensec / ctimensec), number of links (linkN), unique enumeration number (cookie), and version number (versionNum). It also includes a collection of inodeDataLists used to record the identifiers of stored data blocks, which stores the names of multiple FileNames or chunkFiles. File metadata Inodes are stored in the RocksDB database on the index disk as key-value pairs. The key is a string consisting of the symbol "(", the disk virtual node number (vnode), the bucket name (bucket), and the file index number (nodeId). The value is the content of the Inode itself.

[0094] Object data block (FileMeta): A Java instance used to store data blocks, including the disk name (lun), data block name (FileName), data block MD5 value (etag), data block size (size), and data block starting position (offset). FileMeta is stored in the RocksDB database on the data disk as a key-value pair. The key is a string consisting of the "#" symbol, the disk virtual node number (vnode), the bucket name (bucket), a SHA1 string, and a 32-bit random string. The value is the content of the FileMeta itself.

[0095] Single data block identifier in object metadata: The data block name (FileName) of a single object data block is recorded in the object metadata to record the data block location.

[0096] The segment object data block identifier (PartInfo) in object metadata is an abstract data block identifier. Each PartInfo identifies a piece of data, including the size, bucket name, MD5 value, object name, version number, and starting position of the segment block. Multiple PartInfo segment blocks form a PartInfo data, which is stored in the metadata of the segment object. The actual data block storage is still stored in FileMeta.

[0097] A ChunkFile is a file segment data block (ChunkFile): It records metadata about a portion of a large file's data blocks. This includes the file index number (nodeId), bucket name (bucket), file name (objName), version number (versionNum), the size of the recorded data block (size), and a chunkList that records the data blocks contained in the ChunkFile. Each chunk in the chunkList is an abstract concept used to record the smallest data block in the file, similar to FileMeta and stored on disk as FileMeta. ChunkFiles are stored in the RocksDB database on the index disk as key-value pairs. The key is a string consisting of the symbol ")", the disk virtual node number (vnode), the bucket name (bucket), and the chunkKey (a string generated by the chunkFile). The value is the content of the ChunkFile itself. It should be noted that a ChunkFile is the segment file mentioned above, such as the first segment file, the second segment file, and the third segment file.

[0098] Write Cache: This is used to temporarily store uploaded file data blocks and accelerate writes. When a data block in the cache reaches 16MB, it is flushed, the corresponding FileMeta is recorded on the data disk, and the chunkFile name of the data block is added to the file metadata inode.

[0099] File operation request (opt): File operations in Linux systems are usually composed of a combination of different operations. Taking the NFS file system as an example, the opt when uploading data includes lookup (find inode) → create (create inode) → setattr (modify metadata attributes) → write (upload data block) → commit (flush data block) → ... → ... → write → commit.

[0100] Hole file: When files are uploaded concurrently, the written data blocks may not be arranged in order. If there is a gap between two pieces of data, it needs to be filled with a hole file so that the gap can be filled and overwritten with data when the next data is available.

[0101] To reduce the complexity of compatible solutions and avoid the performance loss caused by request translation, this paper designs a multi-level, segmented file metadata storage mechanism, enabling object storage systems to directly parse and process operation requests from network file systems. Furthermore, based on this compatible format, this paper further optimizes large file download performance by proposing strategic background data aggregation and file download pre-read caching solutions. A detailed description is provided below.

[0102] First, we introduce the multi-level segmented file metadata storage mechanism.

[0103] In an object storage system, each piece of data is encapsulated as an independent "object". An object includes metadata for recording the data attributes (such as object name, time, size, version number, bucket name, data name) and data blocks for recording the actual object data. Both are stored in the index disk and data disk in the form of key-value pairs. When a user needs to perform data operations on an object, the data of the object will be obtained through the unique identifier of the object (usually a string index composed of information such as the object name and bucket name). For example, when calling the getObject interface to download an object, the metadata (MetaData) of the object is first obtained from the index disk through a string composed of the object name of the requested object and the bucket name where it is located, and then the actual data is read based on the data block information (FileName) parsed from the obtained metadata.

[0104] While the data storage locations of file systems differ significantly from those in object storage systems, they maintain a similar metadata / data block structure—that of files. Whenever a file is operated on, the inode information corresponding to the file's nodeId is retrieved, and then the corresponding data blocks in the disk data partition are located for reading or writing. This fundamentally shares commonality with the operational logic of object storage systems. This similar metadata / data block organization logic provides a theoretical basis for compatible file data with object storage, allowing file metadata to be stored in the same manner on index disks.

[0105] In MacroShan's object storage system, the formats of object data blocks are mainly divided into two categories: regular objects and segmented objects. Among them, regular objects refer to objects whose object metadata only records a complete data block name. When downloading such objects, you can directly index the FileMeta of the corresponding data disk according to the FileName in MetaData to read the object; for segmented objects, their MetaData records a data block array PartInfo[] containing multiple segments of data. Each segment block records the corresponding data block size, the starting position of the data block reading, and the data block index name. The status of each segment block is equal to that of a regular object data block, corresponding to the FileMeta mentioned above. When downloading data, objects are usually read from each data disk concurrently through data streams (flatMap technology in Java stream programming).

[0106] Compared to object storage, file data blocks naturally conform to the data storage model of segmented objects. File uploads are usually the result of a combination of multiple opts. Taking the currently widely used NFS network file as an example, data upload requires the steps of lookup→create→setattr→write→commit. Actual data writing usually requires multiple write operations. The amount of data carried by each request does not exceed the size specified by the Linux kernel and is not necessarily equal. There may be jumps (holeFiles) and overwrites at the start and end positions. Therefore, file data can be recorded by segmented splicing. At the same time, the segmented data can be converted into PartInfo[] when the object storage system reads the file, allowing the file data to be read and written through the object system.

[0107] Specifically, in the file data format scheme designed by the present invention, a data block of a file can have at most three types when recorded in metadata, such as Figure 2 shown.

[0108] Figure 2 The following is a schematic diagram of the three forms in which the data of a file exists in metadata provided in this specification.

[0109] from Figure 2As can be seen, a file data can include two types of MetaData. If the file is a large file, then multiple FileNames are recorded in the MetaData corresponding to the file; if the file is a small file, then one FileName is recorded in the MetaData corresponding to the file; if the data of the file is initially passed in by the object side, when the file side subsequently appends data to the file, the file will be converted from MetaData to Inode format, and the appended data block will be written to inodeDataList in the form of chunkFile.

[0110] In actual applications, the specific data block recording operations will vary depending on the source of the data block and the capacity of the storage location, and are introduced one by one as follows.

[0111] 1) When all uploaded data blocks are passed in from the file side: Whenever the data blocks in the write cache (WriteCache) accumulate to 16MB, a flush is completed. If there is no data in the Inode at this time, or the previous chunkFile has reached the capacity limit of 5G, or the number of chunks in the chunkFile has reached the upper limit of 1024, a new chunkFile is generated, and the data block information of the data block to be stored (i.e., chunk) is saved in the chunkList of the chunkFile. The chunkFile needs to be recorded in the index disk with a key starting with the ")" symbol, and the chunkFile is saved in the inodeDataList of the Inode at the same time.

[0112] If the chunkFile at the location corresponding to the currently flushed data block has not reached capacity, or the number of chunks in the chunkFile has not reached the upper limit, the data block information is saved to the chunkFile at the corresponding location and the record in the index disk is updated. If overwriting or skipping is required, the existing data blocks in the chunkList are trimmed or a holeFile is added to fill the hole.

[0113] For a large file, the multiple chunkFiles it contains will be recorded in the inodeDataList of the inode. Each inodeData in the inodeDataList is a chunkFile. Each chunk in the chunkList list in each chunkFile has the same status as the FileName in the object data block, and also contains the index name of the disk where the data is located, the data block size, and the starting position of the data block reading.

[0114] 2) When the uploaded data block is converted from object data: regardless of whether the source object is a segmented object, the FileName or PartInfo[] in its data block will be converted into inodeData format, that is, if the source object contains only a single data block, the FileName data block recorded in MetaData will be directly converted into an inodeData data in the inodeDataList in Inode; if the source object is a segmented object, each segmented data block in the PartInfo[] array will be converted into inodeData data.

[0115] During the conversion, the data block information (FileName or PartInfo[]) originally recorded in the object metadata MetaData will be transferred to the corresponding Inode. If the file format Inode does not exist for the object, a new Inode will be created for it.

[0116] 3) When the data uploaded by the object side is overwritten or appended by the data uploaded by the file: first, the FileName or PartInfo in the object metadata MetaData is converted to inodeData, and the corresponding Inode metadata is generated in the index disk.

[0117] For subsequent overwritten or appended data blocks, a new chunkFile will be created based on the location where the data block is uploaded, and the overwritten or appended data block information will be stored in the chunkList in the chunkFile. At this time, the inodeDataList in the metadata inode of a file will contain both FileName and chunkFile.

[0118] For this type of data, when downloading file data, first determine whether the data block name in inodeData carries the chunkFile symbol. If so, obtain the chunkList corresponding to chunkFile from the index disk again. Otherwise, treat it as FileName and read the data directly from the data disk.

[0119] 4) When the data uploaded on the file side is appended by the data uploaded on the object side: The situation is the same as 1). Since overwriting the object data is equivalent to re-uploading an object, the file side data can only be appended on the object side. When appending, first determine whether the capacity of the last chunkFile recorded by the file Inode is full. If not, the newly uploaded data block is recorded in the chunkFile. If it is full, a new chunkFile is created to record it, and the Inode metadata is updated synchronously.

[0120] At the same time, in the data format proposed by the present invention, there are the following points to note:

[0121] 1) For an object, there must be a MetaData. If the object has not been subjected to file-side operations, there is no Inode or chunkFile metadata, and its data blocks are all stored in MetaData. If the object has been subjected to file-side operations, the corresponding Inode and chunkFile metadata are generated, and the data in MetaData is transcribed to Inode.

[0122] For example, when a client uses the file-side interface to request object data from the server, taking file downloading as an example, it usually includes operations such as lookup, open, read, and close (the client needs to use the network file system protocol to access data from the server. The above lookup, open, read, and close are all called opts of the network file system protocol. The download of the client's local file system will trigger multiple opts when called, thus forming a download interface).

[0123] When the client sends a search request through the network file system protocol, the server can generate a corresponding key containing the file name (specifically in the form of vnode / bucket / file path name) according to the file name carried in the request, and search whether the key exists in the current index disk. If so, it means that the data to be downloaded exists. If not, the data to be downloaded does not exist.

[0124] Furthermore, if the key exists, the MetaData (metadata in object format) of the data to be downloaded is obtained, and it is determined whether the nodeId containing the identification file format in the MetaData has a value. If the nodeId has a value (that is, the value is greater than 0), the key corresponding to the corresponding Inode is generated (such as vnode / bucket / nodeId), the inode is found and returned; if nodeId = 0, that is, the current data does not have an inode format, a new nodeId value is generated from the server node requested by the client, and all FileNames in the MetaData or all FileNames contained in partInfo[] are traversed, and the traversed ones are returned. The fileName is put into a chunkFile to be created, and the number of traversed fileNames is counted and the traversed size is accumulated. When every 1024 FileNames are reached or the accumulated size reaches 5G, a new chunkFile is created. When the traversal is completed, a request to create a chunkFile is sent to all distributed server nodes. At this time, all distributed server nodes will store all chunkFiles in the index disk, and remove the FileName data block information recorded in the original MetaData, and store all chunkFiles in the inodeDataList of the inode, thereby completing the conversion to complete the task of downloading files.

[0125] 2) When a file is created, MetaData and Inode are generated simultaneously. If data is uploaded to it, a corresponding chunkFile is generated. When the data passed from the file side is read by the object side, the data blocks in the Inode must be converted to object data blocks, but the metadata records in the index disk will not be changed.

[0126] 3) For the same Inode, if a hard link is created, each hard link corresponds to a separate MetaData. All hard links share the same nodeId and Inode, but have different cookies for easy identification during listing.

[0127] Secondly, strategic background data aggregation is introduced.

[0128] Whenever a read operation is performed through the file system to download data, it is necessary to obtain the file's Inode, and then query the chunkFile in the index disk according to the chunkFile name recorded in the Inode, and finally obtain the bytes of the corresponding data block according to the chunkList recorded in the chunkFile. Usually, for a large file, there are multiple chunkFiles in its Inode, and the number of chunks in the chunkFile is relatively large. To complete the download of a large file, it is necessary to repeatedly obtain the file's metadata many times, which makes the download affected by network latency and computing overhead amplified, thereby reducing the reading performance of large files. In this regard, this manual also designs a background data block aggregation mechanism, which can re-read the stored files according to the preset merging strategy, merge small data blocks, and then write them to the disk, thereby achieving the purpose of reducing the number of metadata reads and improving download performance.

[0129] Specifically, there are two types of background aggregation strategies mentioned above: one is the business priority strategy, which ensures that the foreground business performance is maintained at more than 90% of the level when the background aggregation mechanism is not enabled. At this time, data block aggregation is performed with a concurrency of 1 (the number of chunkFiles merged at the same time is 1), and a single chunkFile occupies up to 128MB of cache. After the aggregation is completed, the number of chunks in the metadata can be reduced to 1 / 8 of the original; the other is the aggregation priority strategy, which is suitable for execution when the foreground traffic is small, ensuring that the number of small data blocks recorded in the metadata is minimized. At this time, data block aggregation is performed with a concurrency of 16 (the number of chunkFiles merged at the same time is 16), and a single chunkFile occupies up to 128MB of cache. After the aggregation is completed, the number of chunks in the metadata can be reduced to 1 / 8 of the original.

[0130] In the above description of the background aggregation method, the concurrency refers to the number of chunkFile merges executed simultaneously by the current node. In both strategies, the cache occupied by each chunkFile during the merge is 128MB, which can accommodate up to 8 16MB chunk data blocks. Therefore, the amount of chunk data can be reduced by 87.5% after the merge. According to the above strategy, the process of implementing background data aggregation is as follows: Figure 3 shown.

[0131] Figure 3 This is a flowchart of a background data aggregation mechanism provided in this manual.

[0132] The specific steps are as follows:

[0133] 1) File Scanning: Create a scanner to scan the inode metadata prefixed with "(" stored on each node's index disk. Only files whose nodeId modulo the value equals the current node's index value are merged. For example, in a three-node storage system, the node index numbers are N = 0, 1, 2. If the result of nodeId % 3 equals the current node's index number and there is a chunkfile with more than 200 chunks, the metadata inode corresponding to that nodeId is passed to the queue QueueInode (the queue has a capacity of 100), ensuring that the same file is aggregated in only one node. If the queue is full, the current scan position is recorded and the scan is exited. Wait until all inodes in the queue have aggregated 50% of the data blocks before reactivating the scanner and continuing from the recorded position.

[0134] 2) Aggregation Mark: The metadata passed into the queue will be received by the queue's subscriber Merger. Before aggregating the data blocks, a file marking request must first be sent to all distributed nodes, and the aggregation flag isMerge in the metadata Inode of the current file to be aggregated is set to true, indicating that the file is being merged.

[0135] If there is new data to be written at this time, a merge rollback processing request will be sent to the node performing aggregation before executing data writing, so that it will immediately stop the data aggregation of the current file, asynchronously delete some of the aggregated data blocks, and throw the currently processed Inode back to the end of the QueueInode queue to wait for the next processing, thereby ensuring the data consistency of the file being written.

[0136] 3) Data aggregation: When the subscriber Merger receives the inode to be aggregated, it will read the chunkFile from the inode for aggregation according to the currently configured aggregation strategy and the determined concurrency. During aggregation, the md5 value of the data block to be aggregated is first calculated for subsequent comparison. Then, each chunkFile is sequentially read from the data disk into a 128MB cache. When the cache is full, a flush is performed so that the chunk can be written to the data disk in 128MB size. The re-write status of the chunkFile is recorded in the index disk. The chunkFile at this time is recorded with a "<" prefix, which is only one prefix different from the original. After the chunkFile is aggregated, it is necessary to calculate whether the md5 value of the newly aggregated data block is equal to the original. If there is a difference, the aggregated data is deleted and thrown back into the queue for the next execution; if there is no difference, the original chunkFile record is deleted in the form of a transaction, and the new "<" record is changed to a ")" record, thereby completing the metadata replacement. In addition, if the chunkFile read in exceeds 128MB when merging data, the last data chunk will be placed in the next aggregation chunk for execution.

[0137] 4) Mark removal: For the Inode that has completed aggregation, a request needs to be sent to all distributed nodes to change the aggregation flag isMerge in the Inode to false.

[0138] 5) Exception handling: To avoid data inconsistency, when the system encounters abnormal situations such as node offline or disk offline, the currently executing aggregation operation will be terminated immediately, and the aggregation will be continued only when the environment is restored and the data and metadata of the three nodes of the file to be aggregated are consistent and there are no abnormalities.

[0139] Finally, the file download pre-reading mechanism is introduced.

[0140] The background data aggregation mechanism can effectively reduce the number of data blocks recorded in the chunkFile, but it cannot reduce the number of times the chunkFile is read from the file metadata Inode. Therefore, when the number of chunkFiles in the Inode is large, downloading files cannot avoid frequent query operations. To this end, the present invention adds a pre-read cache mechanism to the file read operation, such as Figure 4 shown.

[0141] Figure 4 This is a flowchart of a pre-read cache mechanism provided in this manual.

[0142] The specific description is as follows:

[0143] 1) Create / Acquire a Pre-Read Cache Instance: When executing a read operation, create a ReadCache instance, store the corresponding objName and nodeId, and initialize a linked list of chunkFile name-chunkFile instance tuples. When the Pre-Read Cache instance corresponding to the nodeId is first created, it is saved in the public Map set CacheMap, with the key being nodeId and the value being ReadCache.

[0144] 2) Filling the cache: First read the chunkFile in the inode, and populate the 2-tuple linked list with the chunkList corresponding to 100 chunkKeys from the index disk at once. If the chunkList corresponding to the current chunkFile does not exist in the cache, query the index disk again for 100 results and fill it until the reading is complete.

[0145] 3) Get objects from the cache: After completing the cache filling operation, the chunkList required for this time is taken from the cache to read the data block. Then, each time a read operation is received, the chunkList corresponding to the chunkFile will be obtained from the cache first without querying from the index disk. Only when the chunkFile does not exist in the cache will it be re-queried from the index disk.

[0146] 4) Cache invalidation: When writing data, it is first necessary to check whether there is a pre-read cache instance corresponding to the current nodeId in the pre-read cache. If so, delete the instance in time so that it can read the new chunkFile from the index disk again to ensure data consistency.

[0147] Next, the object-file data intercommunication process is introduced.

[0148] This manual will take a 5T file as an example, and combine the above scheme to summarize and describe the file upload, aggregation, download, and intercommunication. The data block is flushed every 16MB through the write cache on the file side. If a 5T file is uploaded through the interface on the file side, 320 chunkFiles will be generated in the end. Each chunkFile contains 1024 chunk data. Each chunkFile is stored in the index disk with ")" metadata. The data block name information corresponding to all chunkFiles is stored in Inode, and the data block information does not exist in MetaData. When reading the 5T object through the interface on the object side, the InodeDataList in Inode is first converted to a PartInfo array. By judging whether the data block name prefix of each PartInfo contains ")", the chunkList corresponding to each chunkFile is read from the index disk, and then the data block download is obtained. Thanks to the background aggregation mechanism, in the absence of foreground traffic, the aggregation strategy is set to the aggregation priority strategy, so that the Inode can merge the data blocks of 16 chunkFiles at the same time, and ultimately reduce the number of chunks in each chunkFile to 128. In addition, due to the presence of a pre-read cache on the file side, when reading file data, the cache can be filled when reading the 1st, 101st, 201st and 301st chunkFiles, so that subsequent reads can directly obtain the chunkList corresponding to the chunkFile from the cache without having to query from the index disk, thereby effectively improving the reading performance.

[0149] In this manual, we innovatively propose a multi-level segmented metadata storage mechanism, which enables the file's inodeData and chunkFile to communicate with the object's PartInfo[]. This allows the object system to directly parse, store, and modify the data generated by the file system, reducing translation losses and data security risks caused by the introduction of third-party tools, and compensating for the only disadvantage of object storage data being immutable through data compatibility.

[0150] In addition, this manual designs a new large file background aggregation solution, which aggregates the qualified chunkFiles according to the specified aggregation strategy while ensuring data consistency, thus solving the problem of slow download caused by too many small metadata files.

[0151] In addition, this manual also designs a file-side download pre-read cache. By pre-reading the chunkList corresponding to the chunkFile, the read operation no longer needs to obtain chunk information from the index disk, thereby improving the download speed of files with a large number of chunkFiles.

[0152] It should be noted that the present invention has three key technical points:

[0153] (1) A data format compatible with the file system for object storage is proposed. The file data block information ChunkFile and Chunk mentioned above can be directly converted with the object data block PartInfo[] and the single data block FileMeta, thereby solving the data compatibility and interoperability issues in cases such as file upload and object download, object upload and file download, file upload and object append, and object upload and file modification. At the same time, new MetaData is created for hard links and they share the same Inode, solving the duplication problem during enumeration.

[0154] (2) A strategic background data aggregation mechanism is proposed to reduce the number of reads. By taking the modulus of the nodeId, the data block aggregation operation of the same file is ensured to be performed only on the same node; when scanning the metadata, only the file Inodes that meet the aggregation requirements are thrown into the queue. After the aggregator Merger receives the Inode, it first sets its aggregation flag to true. Only when data is rewritten, the aggregated data is rolled back and the Inode is thrown back to the queue for the next execution; when aggregating, the number of chunkFile concurrency is set according to the strategy; after the aggregation is completed, the data block md5 is checked for consistency. If not, the aggregated data is rolled back and thrown back to the queue for the next execution; if consistent, the chunkFile is replaced in the form of a transaction, and the aggregation flag in the Inode is set to false after the aggregation is completed.

[0155] (3) A file-side download pre-reading mechanism is proposed to improve download performance. When a read operation is first executed, a pre-read cache is created and the chunkList corresponding to 100 chunkFiles is filled. Subsequent read operations will directly read the chunkList from the cache. Only when the corresponding chunkFile does not exist in the cache, 100 chunkFiles are read from the index disk again to fill the cache until the reading is completed.

[0156] The above is one or more data processing methods implemented in this specification. Based on the same idea, this specification also provides corresponding data processing devices, such as Figure 5 shown.

[0157] Figure 5 A schematic diagram of a data processing device provided in this specification, including:

[0158] The file storage module 501 is configured to, for a current data block being stored, save data block query information for querying the current data block into a data block information list in a first segmented file if the current data block is input from a file end, and record a segmented file identifier of the first segmented file into an index node corresponding to a target file to which the first segmented file belongs; the first segmented file is recorded in the index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data;

[0159] The object storage module 502 is used to store the current data block, if the current data block is obtained by converting the object input from the object end, save the data block query information for querying the current data block in the object metadata into the data block information list in the second segmentation file according to the object metadata of the object, and record the segmentation file identifier of the second segmentation file into the index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0160] Optionally, the object storage module 502 is also used to create a third segmented file if the current data block is obtained by converting the object input from the object side and is appended by the file data block uploaded from the file side, save the data block query information used to query the file data block into the data block information list in the third segmented file, and record the segmented file identifier of the third segmented file into the index node corresponding to the object; the third segmented file is recorded in the index disk; the file data block is one of the data blocks into which the appended data is divided.

[0161] Optionally, the file storage module 501 is also used to, if the current data block is passed in by the file end and is appended to the object uploaded by the object end, if the capacity of the last segmented file of the target file is less than the set capacity, save the data block query information in the object metadata of the appended object for querying each data block in the appended object into the data block information list in the last segmented file according to the object metadata of the appended object; the appended object is the object corresponding to the appended data uploaded by the object end.

[0162] Optionally, the file storage module 501 is also used to create object metadata of the target file if the current data block is passed in from the file end, and record the object metadata of the target file in the index disk; the object metadata of the target file stores the index node identifier of the index node corresponding to the target file.

[0163] Optionally, the file storage module 501 is also used to determine each index node through the created scanner and pass the each index node into the aggregation queue; for each index node in the aggregation queue, according to the order in which the each index node is passed into the aggregation queue, the index node is received through the created aggregator, and the aggregation flag position of the index node is set to the first mark, and the first mark is used to indicate that the data corresponding to the index node is undergoing aggregation processing; read each segmented file in the index node; for each segmented file in the index node, according to the data block information list of the segmented file, read the data corresponding to the segmented file from the data disk into a preset cache; when the data stored in the preset cache reaches a preset size, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified; the data to be aggregated is the data stored in the preset cache; when each segmented file in the index node completes the aggregation processing, the aggregation flag position of the index node is set to the second mark, and the second mark is used to indicate that the data corresponding to the index node has completed the aggregation processing.

[0164] Optionally, the file storage module 501 is specifically used to determine the md5 value of the data to be aggregated; through the aggregator, the data to be aggregated is stored in the data disk, and the data to be aggregated stored in the data disk is verified according to the md5 value. If the verification passes, the data block information list of the segmented file is modified.

[0165] Optionally, the file storage module 501 is also used to, for each index node, create a pre-read cache instance corresponding to the index node when the file end reads the data corresponding to the index node for the first time; query a preset number of segmented files in the index node from the index disk to obtain the segmented file identifiers of the segmented files and the data block information list of the segmented files; fill the segmented file identifiers of the segmented files and the data block information list of the segmented files into the pre-read cache instance corresponding to the index node; and read the data corresponding to the index node according to the pre-read cache instance corresponding to the index node.

[0166] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 A data processing method is provided.

[0167] This manual also provides Figure 6 The one shown corresponds to Figure 1 Schematic diagram of the electronic equipment. Figure 6As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0168] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always program the improved process flow into the hardware circuit to obtain the corresponding hardware circuit structure. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0169] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0170] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0171] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0172] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0177] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0178] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0179] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0180] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0182] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0183] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A data processing method, characterized in that: The method includes: For the current data block stored, If the current data block is input from the file end, data block query information for querying the current data block is saved in a data block information list in the first segment file, and a segment file identifier of the first segment file is recorded in an index node corresponding to a target file to which the first segment file belongs; the first segment file is recorded in an index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data; If the current data block is obtained by converting the object input from the object side, data block query information for querying the current data block in the object metadata is saved in a data block information list in a second segmentation file according to the object metadata of the object, and the segmentation file identifier of the second segmentation file is recorded in an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block obtained by object conversion is one of the data blocks into which the data corresponding to the object is divided.

2. The method according to claim 1, wherein The method further comprises: If the current data block is obtained by converting the object input from the object side and is appended to the file data block uploaded from the file side, a third segmented file is created, data block query information for querying the file data block is saved in a data block information list in the third segmented file, and a segmented file identifier of the third segmented file is recorded in the index node corresponding to the object; the third segmented file is recorded in the index disk; the file data block is one of the data blocks into which the appended data is divided.

3. The method according to claim 1, wherein The method further comprises: If the current data block is passed in by the file side and is appended to the object uploaded by the object side, and if the capacity of the last segment file of the target file is less than the set capacity, data block query information for querying each data block in the appended object in the object metadata of the appended object is saved in the data block information list of the last segment file based on the object metadata of the appended object; the appended object is the object corresponding to the appended data uploaded by the object side.

4. The method according to claim 1, wherein The method further comprises: If the current data block is transferred from the file end, the object metadata of the target file is created and recorded in the index disk; the object metadata of the target file stores the index node identifier of the index node corresponding to the target file.

5. The method according to claim 1, wherein The method further comprises: Determine each index node through the created scanner and pass the index node into the aggregation queue; For each index node in the aggregation queue, the index node is received by the created aggregator in the order in which the index nodes are passed into the aggregation queue, and the aggregation flag position of the index node is set to a first flag, where the first flag is used to indicate that the data corresponding to the index node is undergoing aggregation processing; Read each segment file in the index node; For each segmented file in the index node, the data corresponding to the segmented file is read from the data disk into a preset cache according to the data block information list of the segmented file. When the data stored in the preset cache reaches a preset size, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified; the data to be aggregated is the data stored in the preset cache; When aggregation processing is completed for each segment file in the index node, the aggregation flag position of the index node is set to a second flag, and the second flag is used to indicate that the data corresponding to the index node has completed aggregation processing.

6. The method according to claim 5, wherein The data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified, specifically including: Determine the md5 value of the data to be aggregated; The data to be aggregated is stored in the data disk through the aggregator, and the data to be aggregated stored in the data disk is verified according to the md5 value. If the verification passes, the data block information list of the segmented file is modified.

7. The method according to claim 1, wherein The method further comprises: For each index node, when the file end reads the data corresponding to the index node for the first time, a pre-read cache instance corresponding to the index node is created; Querying a preset number of segment files in the index node from the index disk to obtain segment file identifiers of the segment files and a data block information list of the segment files; Filling the segment file identifier of each segment file and the data block information list of each segment file into the pre-read cache instance corresponding to the index node; According to the pre-read cache instance corresponding to the index node, the data corresponding to the index node is read.

8. A data processing device, characterized in that: The device includes: The file storage module is configured to, for a current data block stored, save data block query information for querying the current data block into a data block information list in a first segmented file if the current data block is input from a file end, and record a segmented file identifier of the first segmented file into an index node corresponding to a target file to which the first segmented file belongs; the first segmented file is recorded in an index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data; The object storage module is used to store a current data block, if the current data block is obtained by converting an object input from an object end, save data block query information for querying the current data block in the object metadata into a data block information list in a second segmentation file according to the object metadata of the object, and record the segmentation file identifier of the second segmentation file into an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block obtained by object conversion is one of the data blocks into which the data corresponding to the object is divided.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images