Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. The method comprises the following steps: for a stored current data block, if the current data block is transmitted by a file end, storing data block query information for querying the current data block into a data block information list in a first segmented file, recording the segmented file identifier of the first segmented file into an index node corresponding to a target file to which the first segmented file belongs; if the current data block is obtained by converting an object transmitted by an object end, storing data block query information used for querying the current data block in object metadata into a data block information list in a second segmented file according to the object metadata of the object, and recording the segmented file identifier of the second segmented file into the index node corresponding to the object.

Description

Technical Field

[0001] The present invention relates to the field of data storage, and in particular to a data processing method, device, storage medium and electronic device. Background Art

[0002] In recent years, with the rapid development of the Internet and the explosive growth of information data, the attention paid to data storage has been increasing. Traditional centralized file system storage solutions have been gradually replaced by distributed storage systems, including: Hadoop Distributed File System (HDFS), which has the advantages of high fault tolerance and low-cost deployment, and can provide high-throughput access to application data; Google File System (GFS), which adopts a central server model and increases storage capacity through master-slave node division while ensuring metadata consistency; and General Parallel File System (GPFS), which can improve the recovery speed of single-point failure of large-scale storage data through cross-node replication of data. Distributed storage solutions represented by file systems have occupied a major position in enterprise big data storage and analysis. However, with the widespread application of artificial intelligence technology and the popularization of various intelligent terminals, massive unstructured data of different sizes and with more reads and less writes are generated and processed, making the file system unable to meet the market's growing demand for unstructured data storage due to shortcomings such as high expansion cost, limited capacity, and inconvenience of sharing.

[0003] Amazon Simple Storage Service (Amazon S3) is an object storage solution launched by Amazon to solve the above problems. It manages metadata and data blocks through "objects", flattens the storage structure by indexing objects with globally unique identifiers, and simplifies data management. In Amazon S3, users can store and retrieve any amount of object data according to a series of HTTP RESTful interfaces provided by the S3 protocol. The generated object metadata and data blocks are stored on different disks of each distributed node according to different redundancy strategies. When expanding, the cluster can be quickly expanded through replication (replica strategy) or EC coding (erasure code strategy) according to the redundancy strategy of the hardware device where the data (metadata) is located. At the same time, the introduction of bucket hashing (bucket sharding) technology enables the object storage solution to theoretically achieve unlimited expansion of the capacity of a single storage bucket and break through the performance bottleneck of reading and writing.

[0004] Compared with distributed file systems, distributed object storage has the characteristics of higher scalability, lower cost, and larger storage capacity, and is naturally suitable for the storage of unstructured data. However, it is inevitable that due to its flat storage structure and unique objects, the performance of modifying data after writing is poor, and it cannot cope with high-time business. In this regard, the intercommunication between object storage and file systems through gateway conversion such as S3FS has become a mainstream technology in existing solutions. This solution can translate file write operations into object storage upload requests based on FUSE (Filesystem in Userspace) technology, thereby achieving data intercommunication. However, due to the essential differences in the storage structures of the two, any data conversion operation will cause a large loss in read and write performance, and the increase in upload links will also bring the risk of data inconsistency. Therefore, constructing a data format compatible with object storage and file systems has become the key to solving the above problems. Summary of the invention

[0005] This specification provides a data processing method, device, storage medium and electronic device to partially solve the above problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a data processing method, which includes:

[0008] For the current data block stored,

[0009] If the current data block is input from the file end, data block query information for querying the current data block is saved in a data block information list in the first segment file, and a segment file identifier of the first segment file is recorded in an index node corresponding to a target file to which the first segment file belongs; the first segment file is recorded in an index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data;

[0010] If the current data block is obtained by converting an object transmitted from the object end, the data block query information used to query the current data block in the object metadata is saved in a data block information list in a second segmentation file according to the object metadata of the object, and the segmentation file identifier of the second segmentation file is recorded in an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0011] Optionally, the method further comprises:

[0012] If the current data block is obtained by converting the object transmitted from the object end and is appended by the file data block uploaded from the file end, a third segmented file is created, data block query information used to query the file data block is saved in a data block information list in the third segmented file, and the segmented file identifier of the third segmented file is recorded in the index node corresponding to the object; the third segmented file is recorded in the index disk; the file data block is one of the data blocks into which the appended data is divided.

[0013] Optionally, the method further comprises:

[0014] If the current data block is passed in by the file side and is appended by the object uploaded by the object side, if the capacity of the last segment file of the target file is less than the set capacity, according to the object metadata of the appended object, the data block query information used to query each data block in the appended object in the object metadata of the appended object is saved in the data block information list of the last segment file; the appended object is the object corresponding to the appended data uploaded by the object side.

[0015] Optionally, the method further comprises:

[0016] If the current data block is transferred from the file end, the object metadata of the target file is created and recorded in the index disk; the object metadata of the target file stores the index node identifier of the index node corresponding to the target file.

[0017] Optionally, the method further comprises:

[0018] Determine each index node through the created scanner, and transfer each index node into the aggregation queue;

[0019] For each index node in the aggregation queue, the index node is received through the created aggregator in the order in which the index nodes are transmitted into the aggregation queue, and the aggregation flag position of the index node is set to a first mark, where the first mark is used to indicate that the data corresponding to the index node is undergoing aggregation processing;

[0020] Read each segment file in the index node;

[0021] For each segmented file in the index node, according to the data block information list of the segmented file, the data corresponding to the segmented file is read from the data disk into a preset cache; when the data stored in the preset cache reaches a preset size, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified; the data to be aggregated is the data stored in the preset cache;

[0022] When each segment file in the index node has completed the aggregation process, the aggregation mark position of the index node is set to a second mark, and the second mark is used to indicate that the data corresponding to the index node has completed the aggregation process.

[0023] Optionally, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified, specifically including:

[0024] Determine the md5 value of the data to be aggregated;

[0025] The data to be aggregated is stored in the data disk through the aggregator, and the data to be aggregated stored in the data disk is verified according to the md5 value. If the verification passes, the data block information list of the segmented file is modified.

[0026] Optionally, the method further comprises:

[0027] For each index node, when the file end reads data corresponding to the index node for the first time, a pre-read cache instance corresponding to the index node is created;

[0028] Querying a preset number of segment files in the index node from the index disk to obtain segment file identifiers of the segment files and a data block information list of the segment files;

[0029] Fill the segment file identifier of each segment file and the data block information list of each segment file into the pre-read cache instance corresponding to the index node;

[0030] According to the pre-read cache instance corresponding to the index node, the data corresponding to the index node is read.

[0031] This specification provides a data processing device, the device comprising:

[0032] The file storage module is used for, for a current data block to be stored, if the current data block is transferred from a file end, saving data block query information for querying the current data block into a data block information list in a first segmentation file, and recording a segmentation file identifier of the first segmentation file into an index node corresponding to a target file to which the first segmentation file belongs; the first segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data;

[0033] The object storage module is used for storing a current data block, if the current data block is obtained by converting an object input from an object end, saving data block query information in the object metadata for querying the current data block into a data block information list in a second segmentation file according to the object metadata of the object, and recording the segmentation file identifier of the second segmentation file into an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0034] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0035] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the program.

[0036] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0037] In the data processing method provided in the present specification, for the current data block stored, if the current data block is transmitted from the file end, data block query information for querying the current data block is saved in a data block information list in a first segmentation file, and a segmentation file identifier of the first segmentation file is recorded in an index node corresponding to a target file to which the first segmentation file belongs; if the current data block is obtained by converting an object transmitted from the object end, based on the object metadata of the object, data block query information for querying the current data block in the object metadata is saved in a data block information list in a second segmentation file, and the segmentation file identifier of the second segmentation file is recorded in an index node corresponding to the object.

[0038] It can be seen from the above method that in the data processing method provided in this specification, the data uploaded by the file end can be converted into each other with the data uploaded by the object end, thereby solving the data compatibility and interoperability problems in cases where the data uploaded by the file end is downloaded by the object end, the data uploaded by the object end is downloaded by the file end, the data uploaded by the file end is appended by the object end, and the data uploaded by the object end is modified by the file end. This essentially solves the differences between the two storage structures and greatly improves the read and write performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0040] Figure 1 A flowchart of a data processing method provided in this specification;

[0041] Figure 2 A schematic diagram of three forms of data in a file provided for this specification in metadata;

[0042] Figure 3 A flowchart for implementing a background data aggregation mechanism provided in this specification;

[0043] Figure 4 A flowchart of a pre-reading cache mechanism provided in this manual;

[0044] Figure 5 A schematic diagram of a data processing device provided in this specification;

[0045] Figure 6 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0047] In the prior art, regarding the compatibility and intercommunication issues between S3 object storage and file system data, the existing gateway conversion technology S3FS can write, read, rename, list, delete and other operations on object data in the logic of files, but due to the essential differences between the data structures of "objects" and "files", S3FS has a large performance loss in the translation process of data operation requests. At the same time, the additional system complexity will reduce data security and increase maintenance costs, and it cannot be stably and reliably applied to production environments. In response to this, the present invention designs a file system metadata format that is compatible with existing object storage, which essentially eliminates the huge structural differences between the two, so that the object storage system can directly parse and process operation requests from the network file system, thereby achieving the purpose of simultaneously improving the compatible data read and write performance and data security.

[0048] Based on this, the present specification provides a data processing method, for the current data block stored, if the current data block is passed in by the file end, the data block query information used to query the current data block is saved in the data block information list in the first segmentation file, and the segmentation file identifier of the first segmentation file is recorded in the index node corresponding to the target file to which the first segmentation file belongs; if the current data block is obtained by converting the object passed in by the object end, according to the object metadata of the object, the data block query information used to query the current data block in the object metadata is saved in the data block information list in the second segmentation file, and the segmentation file identifier of the second segmentation file is recorded in the index node corresponding to the object.

[0049] This approach allows data to be processed in a more reliable and performant manner, and data processed by both the file side and the object side can be downloaded by both sides.

[0050] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0051] Figure 1 A flowchart of a data processing method provided in this specification includes the following steps:

[0052] S101: For the current data block stored, if the current data block is transmitted from the file end, save the data block query information used to query the current data block into the data block information list in the first segment file, and record the segment file identifier of the first segment file into the index node corresponding to the target file to which the first segment file belongs.

[0053] In this specification, the execution entity used to implement a data processing method can be a terminal device such as a laptop computer, a tablet computer, or, of course, a server. For the sake of ease of description, this specification only takes the server as an execution entity as an example to illustrate a data processing method provided in this specification.

[0054] For the current data block stored, if the current data block is transferred from the file end, the server saves the data block query information used to query the current data block into the data block information list in the first segment file, and records the first segment file in the index disk. The data block query information used to query the current data block may include the index name of the disk where the data is located, the data block size, and the starting position of the data block reading.

[0055] For example, when storing the first segment file, it can be stored in the RocksDB database of the index disk in the form of a key-value pair, where the key name (key) is a string composed of the symbol ")", the disk virtual node number (vnode), the bucket name (bucket), and the chunkKey (a string generated by the first segment file), and the value (value) is the content of the first segment file itself.

[0056] It should be noted that when storing the target data corresponding to the target file, the data block query information of multiple data blocks can be recorded in the data block information list of the first segmented file. When the capacity of the first segmented file is greater than or equal to the set capacity, a new segmented file needs to be created, and the remaining target data continues to be uploaded to the new segmented file. The capacity of the first segmented file here can refer to the data capacity or the number of data blocks of the first segmented file. For example, when the data capacity of the first segmented file reaches 5G or the number of data blocks reaches 1024, a new segmented file is created.

[0057] Moreover, during the upload process, the first segment file is not recorded in the index disk until the data block information list of the first segment file reaches the set capacity, but the first segment file is recorded in real time to ensure that no uploaded data block is missed to ensure data reliability. The set capacity of the segment file is set so that when reading the file, the read single segment file will not exceed the single frame RSocket data packet due to being too large (wherein RSocket is a network communication technology that can be used for data transmission between distributed nodes).

[0058] Then, the server records the segment file identifier of the first segment file in the index node corresponding to the target file to which the first segment file belongs, wherein the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data.

[0059] For example, when storing index nodes, they can be stored in the RocksDB database of the index disk in the form of key-value pairs. The key name (key) is a string composed of the symbol "(", the disk virtual node number (vnode), the bucket name (bucket), and the file index number (nodeId), and the value (value) is the content of the index node itself.

[0060] That is to say, when the server stores the target data corresponding to the target file, each time the data in the file write cache (WriteCache) accumulates to a preset size (such as 16MB), a flush is completed, where the file write cache (WriteCache) is used to temporarily store the uploaded file-side data blocks and achieve write acceleration.

[0061] If there is no data in the index node corresponding to the target file at this time, or the data blocks stored in the last segment file recorded in the index node have reached the capacity limit (e.g., 5G), or the number of data blocks stored in the last segment file has reached the limit (e.g., 1024), a new segment file is generated, the data block query information of the currently stored data block is saved in the data block information list of the new segment file, and the new segment file is recorded in the index disk, and then the server records the segment file identifier of the new segment file in the index node corresponding to the target file.

[0062] If the last segment file recorded in the index node of the target file has not reached the capacity or quantity limit at this time, the data block query information of the currently stored data block is saved to the data block list of the last segment file, and the record of the last segment file in the index disk is updated.

[0063] Furthermore, if the current data block is passed in from the file end, the server can also create the object metadata of the target file and record the object metadata of the target file in the index disk. For example, if the object metadata is stored, it can be stored in the RocksDB database of the index disk in the form of a key-value pair, where the key name (key) is a string composed of the symbol "*", the disk virtual node number (vnode), the bucket name (bucket), and the object name (objName), and the value (value) is the content of the object metadata itself. Among them, the object metadata of the target file stores the index node identifier of the index node corresponding to the target file, and the index node identifier is the file index number (nodeId) mentioned in the above process.

[0064] Based on this, if the current data block is passed in by the file end and is appended by the object uploaded by the object end, if the capacity of the last segment file of the target file is less than the set capacity, the server saves the data block query information used to query each data block in the object metadata of the appended object to the data block information list of the last segment file according to the object metadata of the appended object; if the capacity of the last segment file of the target file is greater than or equal to the set capacity, the server creates a new segment file, saves the data block query information used to query each data block in the object metadata of the appended object to the data block information list of the new segment file, and updates the index node corresponding to the target file (that is, the segment file identifier of the new segment file is recorded in the index node of the target file). Among them, the appended object is the object corresponding to the appended data uploaded by the object end.

[0065] S102: For the current data block stored, if the current data block is obtained by converting an object input from the object end, according to the object metadata of the object, save the data block query information for querying the current data block in the object metadata into a data block information list in the second segmentation file, and record the segmentation file identifier of the second segmentation file into the index node corresponding to the object.

[0066] If the current data block is converted from an object passed in from the object side, that is, if there is an object passed in from the object side that needs to be operated on by the file side, the server can convert the object passed in from the object side into a format stored by the file side. Specifically, the server saves the data block query information used to query the current data block in the object metadata into a data block information list in the second segment file based on the object metadata of the object, and records the segment file identifier of the second segment file into the index node corresponding to the object, wherein the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0067] It should be noted that the data block query information of multiple data blocks can be stored in the object metadata of an object. When the data corresponding to the object is converted into the storage format of the file end, each data block in the object metadata is traversed, and the data block query information corresponding to each data block is recorded in the second segment file, and the second segment file is recorded in the index disk. At this time, it is queried whether the index node identifier (i.e., nodeId) stored in the object metadata has a value. If there is a value (i.e., nodeId is greater than 0), the segment file identifier of the second segment file is recorded in the index node corresponding to the object; if there is no value (i.e., nodeId = 0), it can be considered that the object does not exist in the storage format of the file end, and a new index node is generated as the index node corresponding to the object, and the segment file identifier of the second segment file is recorded in the new index node. At the same time, it is also necessary to remove the data block query information recorded in the object metadata of the object, and record the index node identifier of the new index node in the object metadata.

[0068] In addition, while recording the data block query information corresponding to each data block in the second segment file, the second segment file is recorded in the index disk. When the capacity of the second segment file is greater than or equal to the set capacity (that is, the capacity of the second segment file can refer to the data capacity or the number of data of the second segment file, such as when the data capacity of the second segment file reaches 5G or the number of data blocks reaches 1024), it is necessary to create a new segment file and continue to record the remaining data blocks in the object into the new segment file.

[0069] Furthermore, if the current data block is obtained by converting the object input from the object side and is appended by the file data block uploaded from the file side, a third segment file is created, the data block query information for querying the file data block is saved in the data block information list in the third segment file, and the third segment file is recorded in the index disk. Then, the server records the segment file identifier of the third segment file in the index node corresponding to the object. The file data block is one of the data blocks into which the appended data is divided.

[0070] The first segment file, the second segment file, and the third segment file mentioned in the above process are all metadata used to store part of the data of a large file, which may include a file index number (nodeId), a bucket name (bucket), a file name (objName), a version number (versionNum), the size of the recorded data block (size), and a data block information list used to record query information of the included data blocks.

[0071] It should be noted that for an object, there must be object metadata corresponding to the object. If the object has not been operated on the file side, there is no index node corresponding to the object, and the data block query information of the object is stored in the object metadata. If the object has been operated on the file side, an index node corresponding to the object is generated, and the data in the object metadata is transcribed into the index node.

[0072] When a file is created, the object metadata and index node corresponding to the file are generated at the same time. If data is uploaded to it, the corresponding segment files are generated. When the file is read by the object end, the data blocks corresponding to the index nodes need to be converted into data blocks corresponding to the object so that the object end can read the file. However, the index nodes and segment files recorded in the index disk will not be changed. Only the data blocks corresponding to the index nodes are temporarily converted into data blocks corresponding to the object.

[0073] It should also be noted that for the same Inode, if a hard link is created, each hard link corresponds to separate object metadata, and all hard links share the same index node identifier (i.e., nodeId), but there are different unique enumeration identifiers (cookies) for easy identification during enumeration.

[0074] For file data stored in the form of index nodes, when reading data from the file end, the server usually needs to query the segmented files recorded in the index node from the index disk according to the segmented file identifier recorded in the index node, and then read the data according to the data block information list of each segmented file. However, if the file data is large, there may be multiple segmented files, and the data block information list of each segmented file has a large number of data blocks. When reading the file data, it is necessary to repeatedly obtain the data block information list of the segmented file from the index disk, which greatly reduces the reading and writing performance. Based on this, the present invention also provides an aggregation processing method, which re-reads the stored files according to a preset merging strategy, and merges small data blocks before writing them to the disk, thereby achieving the purpose of reducing the number of times the data block query information is read and improving the download performance.

[0075] Specifically, a scanner is created to obtain the index nodes prefixed with "(" in the index disk. Since the same index node is stored in each node in a multi-node storage system, in order to aggregate the current index node in only one node, the server can determine the result of the index node identifier (i.e., the value of nodeId) of the index node after taking the modulus. When the result is equal to the file of the current node index number, the data of the current index node is merged. For example, in a three-node storage system, the node index numbers are N=0, 1, 2. If the result of nodeId%3 is equal When the index number of the current node is (e.g., nodeId=7, nodeId%3=1, then the data corresponding to the index node is aggregated only in the first node), and there is a segmented file with more than 200 data blocks in the index node, the index node is passed to the aggregation queue, where the capacity of the aggregation queue is a fixed capacity (e.g., it can accommodate 100 index nodes). If the capacity of the aggregation queue is full, the current scanning position is recorded and the scan is exited, waiting for all index nodes in the aggregation queue to complete 50% of the data block aggregation before re-awakening the scanner to continue scanning from the recorded scanning position.

[0076] For each index node in the aggregation queue, the index node is received through the created aggregator in the order in which the index nodes are passed into the aggregation queue, and the aggregation flag position of the index node is set to the first mark, wherein the first mark is used to indicate that the data corresponding to the index node is undergoing aggregation processing. If new data is written at this time, a merge rollback processing request is sent to the node executing the aggregation before executing the data writing, so that it immediately stops the data aggregation of the current file, and asynchronously deletes some aggregated data blocks, and throws the currently processed index node back to the end of the aggregation queue to wait for the next processing, thereby ensuring the data consistency of the file being written.

[0077] Then, each segmented file in the index node is read. For each segmented file in the index node, according to the data block information list of the segmented file, the data corresponding to the segmented file is read from the data disk into the preset cache. When the data stored in the preset cache reaches a preset size (such as 128MB), the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified, wherein the data to be aggregated is the data stored in the preset cache.

[0078] Specifically, for the index nodes to be aggregated received by the aggregator, each segmented file will be read from the Inode for aggregation according to the currently configured aggregation strategy and the determined concurrency number. For example, there may be two aggregation strategies. One is the business priority strategy: this strategy ensures that the foreground business performance is maintained at more than 90% of the background aggregation that is not enabled. At this time, data block aggregation is performed with a concurrency of 1 (the number of segmented files merged at the same time). A single segmented file occupies up to 128MB of cache. After the aggregation is completed, the number of data blocks in the segmented file can be reduced to 1 / 8 of the original; the other is the aggregation priority strategy: this strategy is suitable for execution when the foreground traffic is small, ensuring that the number of small data blocks recorded in the segmented file is minimized. At this time, data block aggregation is performed with a concurrency of 16. A single segmented file occupies up to 128MB of cache. After the aggregation is completed, the number of data blocks in the segmented file can be reduced to 1 / 8 of the original.

[0079] During aggregation, the md5 value of the data block to be aggregated is first calculated for subsequent comparison, and then the corresponding data blocks in the segmented file are read from the data disk into the preset cache in sequence. When the data stored in the preset cache reaches the preset size (such as 128MB), a flush is performed so that the data corresponding to the segmented file can be written to the data disk with the preset size, and the re-writing status of the segmented file is recorded in the index disk. At this time, the new segmented file is recorded in the index disk with a "<" prefix, which is only different from the original segmented file by a prefix (the original segmented file is recorded in the index disk with a ")" prefix).

[0080] After the segmented file is aggregated, it is necessary to calculate whether the md5 value of the newly aggregated data block is equal to the original md5 value (for example, the original 8 data blocks are aggregated into 1 data block, whether the md5 value of this 1 data block is equal to the md5 value corresponding to the original 8 data blocks). If there is a difference, the aggregated data block is deleted and thrown back into the aggregation queue to wait for the next execution; if there is no difference, the record of the original segmented file is deleted in the form of a transaction, and the "<" record in the new segmented file is changed to a ")" record, thereby completing the replacement of the segmented file. In addition, if the data corresponding to the read-in segmented file exceeds 128MB when the data is merged, the last data block will be placed in the next aggregation block for execution.

[0081] Furthermore, when each segmented file in the index node has completed the aggregation process, the aggregation flag position of the index node is set to the second mark, wherein the second mark is used to indicate that the data corresponding to the index node has completed the aggregation process. To avoid data inconsistency problems, when the system encounters abnormal conditions such as node offline and disk offline, the currently executing aggregation operation is immediately terminated, and the environment is restored. The aggregation of the files to be aggregated is continued when the data and metadata of each node are consistent and there are no abnormalities.

[0082] The background data aggregation mechanism can effectively reduce the number of data blocks recorded in the segmented file, but it cannot reduce the number of times the segmented file is read from the index node. When the number of segmented files in the index node is large, frequent query operations cannot be avoided when downloading files. In this regard, the present invention also provides a pre-reading method to improve the read and write performance.

[0083] Specifically, for each index node, when the file end reads the data corresponding to the index node for the first time, a pre-read cache instance corresponding to the index node is created, and the index node identifier of each index node and the pre-read cache instance corresponding to each index node are stored in a public Map collection in the form of a key-value pair, where the key is nodeId and the value is the pre-read cache instance corresponding to nodeId. Then, the server queries a preset number of segment files in the index node from the index disk, obtains the segment file identifier of each segment file and a data block information list of each segment file, where the preset number can be 100.

[0084] Then, the server fills the segment file identifier of each segment file and the data block information list of each segment file into the pre-read cache instance corresponding to the index node, and reads the data corresponding to the index node according to the pre-read cache instance corresponding to the index node.

[0085] That is to say, when reading the segmented file in the index node for the first time, the data block information lists corresponding to the preset number of segmented file identifiers will be queried from the index disk at one time, and the segmented file identifiers and the data block information lists corresponding to the segmented file identifiers will be filled into the pre-read cache instance. If the data block information list of the segmented file corresponding to the data to be read does not exist in the cache, the data block information lists corresponding to the preset number of segmented file identifiers will be queried from the index disk again and filled until the data to be read is read. After completing the cache filling operation, the data block information list of the segmented file corresponding to the data to be read this time is taken from the cache to read the data block. After that, each time a read operation is received, the data block information list of the segmented file corresponding to the data to be read will be obtained from the cache first, without querying from the index disk. Only when the segmented file does not exist in the cache will it be queried again from the index disk.

[0086] In addition, when data is written, the cache may become invalid. At this time, the server needs to query the pre-read cache to see whether there is a pre-read cache instance corresponding to the index node that is currently writing data. If so, the pre-read cache instance is deleted in time so that it can read the new segment file from the index disk again to ensure data consistency.

[0087] With respect to the method in the above-mentioned specification, a complete embodiment is provided here for detailed description.

[0088] First, in this embodiment, there are some terms, which are explained here.

[0089] Bucket: The basic entity stored in object storage, a container for storing objects.

[0090] Object: The basic entity stored in object storage. An object is a collection of file data and its related attribute information, including the object key, object data, and object metadata.

[0091] Object metadata (MetaData): Object attribute information stored in the index disk, including information such as object name, object creation date, version number, object data size, object bucket name, object account, etc. This data is used to identify the object and record the storage location of the object data through the data block name (FileName or PartInfo array). In the Macroshan MOSS object storage system, MetaData is stored in the RocksDB database of the index disk in the form of key-value pairs. The key name (key) is a string composed of the symbol "*", the disk virtual node number (vnode), the bucket name (bucket), and the object name (objName). The value (value) is the content of the MetaData itself.

[0092] Disk virtual node (vnode): It is any value in the range [0,65535]. It is generated after the disk and the data pool to which the disk belongs are initialized. Each disk has multiple virtual nodes, and a virtual node can only map one disk. The virtual node and consistent hashing technology can reduce the amount of migrated data when the node device is abnormally disconnected or expanded.

[0093] Inode: A data structure used to store file metadata in a file system. In the file data format compatible with MacroShan's object storage, this structure includes attributes such as the file name (name), size (size), unique index number (nodeId), permission (mode), user identification number (uid), group identification number (gid), time (atime / mtime / ctime / atimensec / mtimensec / ctimensec), number of links (linkN), unique enumeration identification number (cookie), and version number (versionNum). It also includes a collection inodeDataList for recording storage data block identifiers, which stores the names of multiple FileName or chunkFile. File metadata Inode is stored in the RocksDB database of the index disk in the form of key-value pairs. The key name (key) is a string composed of the symbol "(", the disk virtual node number (vnode), the bucket name (bucket), and the file index number (nodeId). The value (value) is the content of the Inode itself.

[0094] Object data block (FileMeta): Java instance used to store data blocks, including disk name (lun), data block name (FileName), data block MD5 value (etag), data block size (size), and data block starting position (offset). FileMeta is stored in the RocksDB database of the data disk in the form of key-value pairs. The key name (key) is a string composed of the symbol "#", disk virtual node number (vnode), bucket name (bucket), SHA1 string, and 32-bit random string. The value (value) is the content of FileMeta itself.

[0095] Single data block identifier in object metadata: The data block name (FileName) of a single object data block is recorded in the object metadata to record the data block location.

[0096] Segmented object data block identifier (PartInfo) in object metadata: It is an abstract data block identifier. Each PartInfo identifies a piece of data, including the size, bucket name, md5 value, object name, version number, starting position, etc. of the segmented block. Multiple PartInfo segmented blocks form a PartInfo data that is stored in the metadata of the segmented object. The actual data block storage is still stored in FileMeta.

[0097] File segmentation data block (ChunkFile): Metadata used to record the data block information of a part of a large file, including the file index number (nodeId), bucket name (bucket), file name (objName), version number (versionNum), the size of the recorded data block (size), and chunkList used to record the data block information contained in ChunkFile. Each chunk in chunkList is the smallest abstract concept used to record data blocks in the file, similar to FileMeta, and is also saved as FileMeta on the disk. ChunkFile is stored in the RocksDB database of the index disk in the form of key-value pairs. The key name (key) is a string composed of the symbol ")", the disk virtual node number (vnode), the bucket name (bucket), and the chunkKey (a string generated by chunkFile), and the value (value) is the content of ChunkFile itself. It should be noted that the file segmentation data block is the segmented file mentioned in the above content, such as the first segment file, the second segment file, and the third segment file.

[0098] File write cache (WriteCache): used to temporarily store uploaded file data blocks and accelerate writes. When a data block in the cache is full of 16MB, it is flushed, the corresponding FileMeta is recorded in the data disk, and the chunkFile name of the data segment is added to the file metadata Inode.

[0099] File operation request (opt): Usually, file operations in Linux systems are composed of different operations. Taking the NFS file system as an example, the opt when uploading data includes lookup (find Inode) → create (create Inode) → setattr (modify metadata attributes) → write (upload data block) → commit (flush data block) → ... → ... → write → commit.

[0100] Hole file: When files are uploaded concurrently, the written data blocks may not be arranged in order. If there is a gap between two pieces of data, it is necessary to fill it with a hole file to wait for the next time there is data to fill the hole, that is, to cover the hole with data.

[0101] In order to reduce the complexity of the compatible solution and avoid the performance loss caused by request translation, the present invention designs a multi-level segmented file metadata storage mechanism, so that the object storage system can directly parse and process the operation request from the network file system. In addition, based on this compatible format, the present invention further optimizes the download performance of large files and proposes a strategic background data aggregation and file download pre-reading cache solution. The specific description is as follows.

[0102] First, the multi-level segmented file metadata storage mechanism is introduced.

[0103] In an object storage system, each data is encapsulated as an independent "object". An object includes metadata for recording the data attributes (such as object name, time, size, version number, bucket name, data name) and data blocks for recording the actual object data. Both are stored in the index disk and data disk in the form of key-value pairs. When a user needs to perform data operations on an object, the data of the object will be obtained through the unique identifier of the object (usually a string index composed of information such as the object name and bucket name). For example, when calling the getObject interface to download an object, the metadata (MetaData) of the object is first queried from the index disk through the string composed of the object name of the requested object and the bucket name, and then the actual data is read based on the data block information (FileName) parsed from the obtained metadata.

[0104] Although the data storage location of the file system is significantly different from that of the object storage system, they also maintain similar metadata / data block structures, that is, the metadata / data block structure of the file. Whenever a file is operated, the Inode information corresponding to the nodeId of the file is obtained, and then the data blocks in the corresponding disk data partition are searched for reading and writing. This is essentially similar to the operation logic of the object storage system. The similar metadata / data block organization logic provides a theoretical basis for compatible file data based on object storage, and the metadata of the file can be stored in the index disk in the same way.

[0105] In MacroShan's object storage system, the formats of object data blocks are mainly divided into two categories, regular objects and segmented objects. Among them, regular objects refer to objects whose object metadata only records a complete data block name. When downloading such objects, you can directly index the FileMeta of the corresponding data disk according to the FileName in MetaData to read the object; for segmented objects, their MetaData records a data block array PartInfo[] containing multiple segments of data. Each segment block records the corresponding data block size, the starting position of the data block reading, and the data block index name. The status of each segment block is equal to that of a regular object data block, corresponding to the FileMeta mentioned above. When downloading data, objects are usually read from each data disk concurrently by means of data streams (flatMap technology in Java stream programming).

[0106] Compared with object storage, file data blocks naturally conform to the data storage mode of segmented objects. Among them, file upload is usually the result of a combination of multiple opts. Taking the currently widely used NFS network file as an example, data upload requires steps such as lookup→create→setattr→write→commit. Actual data writing usually requires multiple write operations. The amount of data carried by each request does not exceed the size specified by the Linux kernel, and is not necessarily equal. There are jumps (holeFile) and overwrites at the start and end positions. Therefore, file data can be recorded by segmented splicing, and the segmented data can be converted into PartInfo[] when the object storage system reads the file, so that the file data can be read and written through the object system.

[0107] Specifically, in the file data format scheme designed by the present invention, a data block of a file can have at most three types when recorded in metadata, such as Figure 2 shown.

[0108] Figure 2 The following is a schematic diagram of the three forms in which the data of a file exists in the metadata provided for this specification.

[0109] from Figure 2It can be seen that a file data can include two kinds of MetaData. If the file is a large file, then multiple FileNames are recorded in the MetaData corresponding to the file; if the file is a small file, then one FileName is recorded in the MetaData corresponding to the file; if the data of the file is initially passed in by the object end, when the file end subsequently appends data to the file, the file will be converted from MetaData to Inode format, and the appended data block will be written to inodeDataList in the form of chunkFile.

[0110] In actual applications, the specific data block recording operations will vary depending on the source of the data block and the capacity of the storage location, and are introduced one by one as follows.

[0111] 1) When all the uploaded data blocks are passed in from the file end: Whenever the data blocks in the write cache (WriteCache) accumulate to 16MB, a flush is completed. If there is no data in the Inode at this time, or the previous chunkFile has reached the capacity limit of 5G, or the number of chunks in the chunkFile has reached the upper limit of 1024, a new chunkFile is generated, and the data block information of the current data block (i.e., chunk) to be stored is saved in the chunkList of the chunkFile. The chunkFile needs to be recorded in the index disk with a key starting with the ")" symbol, and the chunkFile is saved in the inodeDataList of the Inode at the same time.

[0112] If the chunkFile at the corresponding position of the data block currently placed on disk has not reached the capacity, or the number of chunks in the chunkFile has not reached the upper limit, the data block information is saved in the chunkFile at the corresponding position and the record in the index disk is updated. If it is necessary to handle overwriting or jumping, the existing data blocks in the chunkList are trimmed or holeFile is added to fill the holes.

[0113] For a large file, the multiple chunkFiles it contains will be recorded in the inodeDataList of the inode. Each inodeData in the inodeDataList is a chunkFile. Each chunk in the chunkList list in each chunkFile has the same status as the FileName in the object data block, and also contains the index name of the disk where the data is located, the data block size, and the starting position for reading the data block.

[0114] 2) When the uploaded data block is converted from object data: regardless of whether the source object is a segmented object, the FileName or PartInfo[] in its data block will be converted into the inodeData format, that is, if the source object contains only a single data block, the FileName data block recorded in MetaData will be directly converted into an inodeData data in the inodeDataList in Inode; and if the source object is a segmented object, each segmented data block in the PartInfo[] array will be converted into inodeData data.

[0115] During the conversion, the data block information (FileName or PartInfo[]) originally recorded in the object metadata MetaData will be transferred to the corresponding Inode. If the file format Inode does not exist for the object, a new Inode will be created for it.

[0116] 3) When the data uploaded by the object end is overwritten or appended by the data uploaded by the file: first, the FileName or PartInfo in the object metadata MetaData is converted to inodeData, and the corresponding Inode metadata is generated in the index disk.

[0117] For the subsequent overwritten and appended data blocks, a new chunkFile will be created according to the location where the data blocks are uploaded, and the overwritten and appended data block information will be stored in the chunkList in the chunkFile. At this time, the inodeDataList in the metadata inode of a file will contain both FileName and chunkFile.

[0118] For this type of data, when downloading file data, first determine whether the data block name in inodeData carries the chunkFile symbol. If so, obtain the chunkList corresponding to chunkFile from the index disk again. Otherwise, treat it as FileName and read the data directly from the data disk.

[0119] 4) When the data uploaded by the file side is appended by the data uploaded by the object: The situation is the same as 1). Since the overwriting operation of object data is equivalent to re-uploading an object, the file side data can only be appended to the object side. When appending, first determine whether the capacity of the last chunkFile recorded by the file Inode is full. If not, the newly uploaded data block is recorded in the chunkFile. If it is full, a new chunkFile is created to record it, and the Inode metadata is updated synchronously.

[0120] At the same time, in the data format proposed by the present invention, there are the following points to note:

[0121] 1) For an object, there must be a MetaData. If the object has not been subjected to file-side operations, there is no Inode or chunkFile metadata, and its data blocks are all stored in MetaData. If the object has been subjected to file-side operations, the corresponding Inode and chunkFile metadata are generated, and the data in MetaData is transcribed to Inode.

[0122] For example, when the client uses the file-side interface to request object data from the server, taking file downloading as an example, it usually includes operations such as lookup, open, read, and close (the client needs to use the network file system protocol to access data from the server. The above lookup, open, read, and close are all called opt of the network file system protocol. The download of the client's local file system will trigger multiple opts when called, thus forming a download interface).

[0123] When the client sends a search request through the network file system protocol, the server can generate a corresponding key containing the file name (in the specific form such as: vnode / bucket / file path name) according to the file name carried in the request, and search whether the key exists in the current index disk. If it exists, it means that the data to be downloaded exists. If not, the data to be downloaded does not exist.

[0124] Furthermore, if the key exists, the MetaData (metadata in object format) of the data to be downloaded is obtained, and it is determined whether the nodeId containing the file format identifier in the MetaData has a value. If the nodeId has a value (that is, the value is greater than 0), a key corresponding to the corresponding Inode (such as vnode / bucket / nodeId) is generated, and the inode is found and returned; if nodeId = 0, that is, the current data does not exist in the inode format, a new nodeId value is generated from the server node requested by the client, and all FileNames in the MetaData or all FileNames contained in partInfo[] are traversed, and the traversed ones are returned. fileName is put into a chunkFile to be created, and the number of traversed fileNames is counted and the traversed size is accumulated. A new chunkFile is created every time 1024 FileNames are reached or the accumulated size reaches 5G. When the traversal is completed, a request to create a chunkFile is sent to all distributed server nodes. At this time, all distributed server nodes will store all chunkFiles in the index disk, and remove the FileName data block information recorded in the original MetaData, and store all chunkFiles in the inodeDataList of the inode, thereby completing the conversion to complete the task of downloading files.

[0125] 2) For a file, MetaData and Inode are generated at the same time when it is created. If data is uploaded to it, the corresponding chunkFile is generated. When the data passed in from the file end is read by the object end, the data block in the Inode needs to be converted into the data block of the object, but the metadata record in the index disk will not be changed.

[0126] 3) For the same Inode, if a hard link is created, each hard link corresponds to a separate MetaData. All hard links share the same nodeId and Inode, but there are different cookies for easy identification during listing.

[0127] Secondly, strategic background data aggregation is introduced.

[0128] Whenever a read operation is performed through the file system to download data, it is necessary to obtain the Inode of the file, and then query the chunkFile in the index disk according to the chunkFile name recorded in the Inode, and finally obtain the bytes of the corresponding data block according to the chunkList recorded in the chunkFile. Usually, for a large file, there are multiple chunkFiles in its Inode, and the number of chunks in the chunkFile is large. To complete the download of a large file, it is necessary to repeatedly obtain the metadata of the file, which makes the impact of network latency and computing overhead on the download amplified, thereby reducing the reading performance of large files. In this regard, a background data block aggregation mechanism is also designed in this manual, which can re-read the stored files according to the preset merging strategy, merge small data blocks and then write them to the disk, so as to achieve the purpose of reducing the number of metadata reads and improving download performance.

[0129] Specifically, there are two types of background aggregation strategies mentioned above: one is the business priority strategy, which ensures that the foreground business performance is maintained at more than 90% of the background aggregation mechanism that is not enabled. At this time, data block aggregation is performed with a concurrency of 1 (the number of chunkFiles merged at the same time is 1), and a single chunkFile occupies up to 128MB of cache. After the aggregation is completed, the number of chunks in the metadata can be reduced to 1 / 8 of the original; the second is the aggregation priority strategy, which is suitable for execution when the foreground traffic is small, ensuring that the number of small data blocks recorded in the metadata is minimized. At this time, data block aggregation is performed with a concurrency of 16 (the number of chunkFiles merged at the same time is 16), and a single chunkFile occupies up to 128MB of cache. After the aggregation is completed, the number of chunks in the metadata can be reduced to 1 / 8 of the original.

[0130] In the above description of the background aggregation method, the concurrency refers to the number of chunkFile merges executed simultaneously by the current node. In both strategies, the cache occupied by each chunkFile during the merge is 128MB, which can accommodate up to 8 16MB chunk data blocks. Therefore, the amount of chunk data can be reduced by 87.5% after the merge. According to the above strategy, the process of implementing background data aggregation is as follows: Figure 3 shown.

[0131] Figure 3 This is a flowchart of a background data aggregation mechanism provided in this specification.

[0132] The specific steps are as follows:

[0133] 1) File scanning: Create a scanner Scanner, scan the Inode metadata with a prefix of "(" stored in the index disk at each node, and merge data only for files whose nodeId value modulo is equal to the current node index value. For example, in a three-node storage system, the node index number is N = 0, 1, 2; if the result of nodeId%3 is equal to the index number of the current node, and there is a chunkFile with more than 200 chunk data blocks, the metadata Inode corresponding to the nodeId is passed to the queue QueueInode (the capacity of the queue is 100), so as to ensure that the same file is aggregated in only one node; if the capacity of the queue is full, the current scanning position is recorded and the scanning is exited, and the scanner is re-awakened to continue execution from the recorded position after all Inodes in the queue have completed 50% of the data block aggregation.

[0134] 2) Aggregation Mark: The metadata passed into the queue will be received by the queue's subscriber Merger. Before aggregating the data blocks, a file mark request must first be sent to all distributed nodes, and the aggregation flag isMerge in the metadata Inode of the current file to be aggregated must be set to true, indicating that the file is being merged.

[0135] If there is new data to be written at this time, a merge rollback processing request will be sent to the node executing aggregation before executing data writing, so that it will immediately stop the data aggregation of the current file, asynchronously delete some of the aggregated data blocks, and throw the currently processed Inode back to the end of the QueueInode queue to wait for the next processing, thereby ensuring the data consistency of the file being written.

[0136] 3) Data aggregation: When the subscriber Merger receives the Inode to be aggregated, it will read the chunkFile from the Inode for aggregation according to the currently configured aggregation strategy and the determined number of concurrencies. During aggregation, the md5 value of the data block to be aggregated is first calculated for subsequent comparison, and then each chunkFile is read from the data disk into a 128MB cache in sequence. When the cache data is full, a flush is performed so that the chunk can be written to the data disk with a size of 128MB, and the re-write status of the chunkFile is recorded in the index disk. At this time, the chunkFile is recorded with a "<" prefix, which is only one prefix different from the original. After the aggregation of the chunkFile is completed, it is necessary to calculate whether the md5 value of the newly aggregated data block is equal to the original one. If there is a difference, the aggregated data is deleted and thrown back to the queue to wait for the next execution; if there is no difference, the original chunkFile record is deleted in the form of a transaction, and the new "<" record is changed to a ")" record, thereby completing the replacement of the metadata. In addition, if the chunkFile read in exceeds 128MB when merging data, the last data chunk will be placed in the next aggregation chunk for execution.

[0137] 4) Mark removal: For the Inode that has completed aggregation, a request needs to be sent to all distributed nodes to change the aggregation flag isMerge in the Inode to false.

[0138] 5) Exception handling: To avoid data inconsistency problems, when the system encounters abnormal situations such as node offline or disk offline, the currently executing aggregation operation will be terminated immediately, and the environment will be restored. Aggregation will not be continued until the data and metadata of the three nodes of the aggregated file are consistent and there are no abnormalities.

[0139] Finally, the file download pre-reading mechanism is introduced.

[0140] The background data aggregation mechanism can effectively reduce the number of data blocks recorded in chunkFile, but it cannot reduce the number of times chunkFile is read from the file metadata Inode. Therefore, when the number of chunkFiles in the Inode is large, frequent query operations cannot be avoided when downloading files. In this regard, the present invention adds a pre-read cache mechanism to the file read operation, such as Figure 4 shown.

[0141] Figure 4 This is a flowchart of a pre-reading cache mechanism provided in this specification.

[0142] The specific description is as follows:

[0143] 1) Create / get a pre-read cache instance: When executing a read operation, create a pre-read cache instance ReadCache, store the corresponding objName and nodeId in it, and initialize a tuple linked list of chunkFile name-chunkFile instance. When the pre-read cache instance corresponding to nodeId is created for the first time, it is saved in the public Map collection CacheMap, with the key being nodeId and the value being ReadCache.

[0144] 2) Fill the cache: Read the chunkFile in the Inode for the first time, and fill the chunkList corresponding to 100 chunkKeys from the index disk into the two-tuple linked list. If the chunkList corresponding to the current chunkFile does not exist in the cache, query 100 results from the index disk again to fill it until the reading is completed.

[0145] 3) Get objects from the cache: After completing the cache filling operation, take the chunkList required this time from the cache to read the data block. Then, each time a read operation is received, the chunkList corresponding to the chunkFile will be obtained from the cache first without querying from the index disk. Only when the chunkFile does not exist in the cache will it be queried from the index disk again.

[0146] 4) Cache invalidation: When writing data, you first need to check whether there is a pre-read cache instance corresponding to the current nodeId in the pre-read cache. If so, delete the instance in time so that it can read the new chunkFile from the index disk again to ensure data consistency.

[0147] Next, the object-file data intercommunication process is introduced.

[0148] This manual will take a 5T file as an example, and summarize and describe the file upload, aggregation, download, and intercommunication in combination with the above scheme. The data block is flushed once every 16MB through the write cache on the file side. If a 5T file is uploaded through the interface on the file side, 320 chunkFiles will be generated in the end. Each chunkFile contains 1024 chunk data. Each chunkFile is stored in the index disk with ")" metadata. The data block name information corresponding to all chunkFiles is stored in Inode, and there is no data block information in MetaData. When reading the 5T object through the interface on the object side, first convert the InodeDataList in Inode into a PartInfo array, and read the chunkList corresponding to each chunkFile from the index disk by judging that the data block name prefix of each PartInfo contains ")", and then obtain the data block download. Thanks to the background aggregation mechanism, in the absence of foreground traffic, the aggregation strategy is set to the aggregation priority strategy, so that the Inode can merge the data blocks of 16 chunkFiles at the same time, and finally reduce the number of chunks in each chunkFile to 128. In addition, due to the presence of a pre-read cache on the file side, when reading file data, the cache can be filled when the 1st, 101st, 201st and 301st chunkFiles are read, so that subsequent reads can directly obtain the chunkList corresponding to the chunkFile from the cache without having to query from the index disk, thereby effectively improving the reading performance.

[0149] In this manual, an innovative multi-level segmented metadata storage mechanism is proposed, which enables the file's inodeData and chunkFile to communicate with the object's PartInfo[], so that the object system can directly parse, store, and modify the data generated by the file system, reducing translation losses and data security risks caused by the introduction of third-party tools, and compensating for the only disadvantage of object storage data that it cannot be changed by means of compatible data.

[0150] In addition, a new large file background aggregation solution is designed in this manual. According to the specified aggregation strategy, the chunkFiles that meet the conditions are aggregated while ensuring data consistency, thereby solving the problem of slow downloading caused by too many small metadata files.

[0151] In addition, this manual also designs a file-side download pre-read cache, which pre-reads the chunkList corresponding to the chunkFile so that the read operation no longer needs to obtain chunk information from the index disk, thereby improving the download speed of files with a large number of chunkFiles.

[0152] It should be noted that the present invention has three key technical points:

[0153] (1) Propose a data format compatible with the file system for object storage. The file data block information ChunkFile and Chunk mentioned above can be directly converted with the object data block PartInfo[] and the single data block FileMeta, thereby solving the data compatibility and interoperability problems in the cases of file upload and object download, object upload and file download, file upload and object append, object upload and file modification, etc.; at the same time, a new MetaData is created for the hard link and the same Inode is shared, solving the duplication problem during enumeration.

[0154] (2) A strategic background data aggregation mechanism is proposed to reduce the number of reads. By taking the modulus of the nodeId, the data block aggregation operation of the same file is ensured to be performed only on the same node; when scanning the metadata, only the file Inodes that meet the aggregation requirements are thrown into the queue. After the aggregator Merger receives the Inode, it first sets its aggregation flag to true. Only when data is rewritten, the aggregated data is rolled back and the Inode is thrown back to the queue to wait for the next execution; when aggregating, the number of chunkFile concurrency is set according to the strategy; after the aggregation is completed, the data block md5 is checked for consistency. If not, the aggregated data is rolled back and thrown back to the queue to wait for the next execution; if consistent, the chunkFile is replaced in the form of a transaction, and the aggregation flag in the Inode is set to false after the aggregation is completed.

[0155] (3) A file-side download pre-reading mechanism is proposed to improve download performance. When a read operation is performed for the first time, a pre-reading cache is created and the chunkList corresponding to 100 chunkFiles is filled; subsequent read operations will directly read the chunkList from the cache. Only when the corresponding chunkFile does not exist in the cache, 100 chunkFiles will be read from the index disk again to fill the cache until the reading is completed.

[0156] The above are one or more data processing methods implemented in this specification. Based on the same idea, this specification also provides corresponding data processing devices, such as Figure 5 shown.

[0157] Figure 5 A schematic diagram of a data processing device provided for this specification includes:

[0158] The file storage module 501 is used for storing a current data block, if the current data block is transferred from the file end, saving the data block query information for querying the current data block into the data block information list in the first segment file, and recording the segment file identifier of the first segment file into the index node corresponding to the target file to which the first segment file belongs; the first segment file is recorded in the index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data;

[0159] The object storage module 502 is used to store the current data block. If the current data block is obtained by converting the object input from the object end, the data block query information used to query the current data block in the object metadata is saved in the data block information list in the second segmentation file according to the object metadata of the object, and the segmentation file identifier of the second segmentation file is recorded in the index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

[0160] Optionally, the object storage module 502 is also used to create a third segmented file if the current data block is obtained by converting the object transmitted from the object end and is appended by the file data block uploaded from the file end, save the data block query information used to query the file data block into the data block information list in the third segmented file, and record the segmented file identifier of the third segmented file in the index node corresponding to the object; the third segmented file is recorded in the index disk; the file data block is one of the data blocks into which the appended data is divided.

[0161] Optionally, the file storage module 501 is also used to, if the current data block is passed in by the file end and is appended by the object uploaded by the object end, if the capacity of the last segmented file of the target file is less than the set capacity, save the data block query information used for querying each data block in the appended object in the object metadata of the appended object into the data block information list in the last segmented file according to the object metadata of the appended object; the appended object is the object corresponding to the appended data uploaded by the object end.

[0162] Optionally, the file storage module 501 is also used to create object metadata of the target file if the current data block is passed in by the file end, and record the object metadata of the target file in the index disk; the object metadata of the target file stores the index node identifier of the index node corresponding to the target file.

[0163] Optionally, the file storage module 501 is also used to determine each index node through a created scanner, and pass the each index node into an aggregation queue; for each index node in the aggregation queue, receive the index node through the created aggregator in the order in which the index nodes are passed into the aggregation queue, and set the aggregation flag position of the index node to a first mark, where the first mark is used to indicate that the data corresponding to the index node is undergoing aggregation processing; read each segmented file in the index node; for each segmented file in the index node, read the data corresponding to the segmented file from the data disk into a preset cache according to the data block information list of the segmented file, and when the data stored in the preset cache reaches a preset size, store the data to be aggregated in the data disk, and modify the data block information list of the segmented file; the data to be aggregated is the data stored in the preset cache; when each segmented file in the index node completes the aggregation processing, set the aggregation flag position of the index node to a second mark, where the second mark is used to indicate that the data corresponding to the index node has completed the aggregation processing.

[0164] Optionally, the file storage module 501 is specifically used to determine the md5 value of the data to be aggregated; through the aggregator, the data to be aggregated is stored in the data disk, and the data to be aggregated stored in the data disk is verified according to the md5 value. If the verification passes, the data block information list of the segmented file is modified.

[0165] Optionally, the file storage module 501 is also used to, for each index node, create a pre-read cache instance corresponding to the index node when the file end reads the data corresponding to the index node for the first time; query a preset number of segmented files in the index node from the index disk to obtain the segmented file identifiers of each segmented file and the data block information list of each segmented file; fill the segmented file identifiers of each segmented file and the data block information list of each segmented file into the pre-read cache instance corresponding to the index node; and read the data corresponding to the index node according to the pre-read cache instance corresponding to the index node.

[0166] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A data processing method is provided.

[0167] This manual also provides Figure 6 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 6As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0168] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0169] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0170] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0171] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0172] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0173] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0177] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0178] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0179] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0180] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0182] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data processing method, characterized in that: The method includes: For the current data block stored, If the current data block is input from the file end, data block query information for querying the current data block is saved in a data block information list in the first segment file, and a segment file identifier of the first segment file is recorded in an index node corresponding to a target file to which the first segment file belongs; the first segment file is recorded in an index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data; If the current data block is obtained by converting an object transmitted from the object end, the data block query information used to query the current data block in the object metadata is saved in a data block information list in a second segmentation file according to the object metadata of the object, and the segmentation file identifier of the second segmentation file is recorded in an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

2. The method according to claim 1, characterized in that The method further comprises: If the current data block is obtained by converting the object transmitted from the object end and is appended by the file data block uploaded from the file end, a third segmented file is created, data block query information used to query the file data block is saved in a data block information list in the third segmented file, and the segmented file identifier of the third segmented file is recorded in the index node corresponding to the object; the third segmented file is recorded in the index disk; the file data block is one of the data blocks into which the appended data is divided.

3. The method according to claim 1, characterized in that The method further comprises: If the current data block is passed in by the file side and is appended by the object uploaded by the object side, if the capacity of the last segment file of the target file is less than the set capacity, according to the object metadata of the appended object, the data block query information used to query each data block in the appended object in the object metadata of the appended object is saved in the data block information list of the last segment file; the appended object is the object corresponding to the appended data uploaded by the object side.

4. The method according to claim 1, characterized in that The method further comprises: If the current data block is transferred from the file end, the object metadata of the target file is created and recorded in the index disk; the object metadata of the target file stores the index node identifier of the index node corresponding to the target file.

5. The method according to claim 1, characterized in that The method further comprises: Determine each index node through the created scanner, and transfer each index node into the aggregation queue; For each index node in the aggregation queue, the index node is received through the created aggregator in the order in which the index nodes are transmitted into the aggregation queue, and the aggregation flag position of the index node is set to a first mark, where the first mark is used to indicate that the data corresponding to the index node is undergoing aggregation processing; Read each segment file in the index node; For each segmented file in the index node, according to the data block information list of the segmented file, the data corresponding to the segmented file is read from the data disk into a preset cache; when the data stored in the preset cache reaches a preset size, the data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified; the data to be aggregated is the data stored in the preset cache; When each segment file in the index node has completed the aggregation process, the aggregation mark position of the index node is set to a second mark, and the second mark is used to indicate that the data corresponding to the index node has completed the aggregation process.

6. The method according to claim 5, characterized in that The data to be aggregated is stored in the data disk, and the data block information list of the segmented file is modified, specifically including: Determine the md5 value of the data to be aggregated; The data to be aggregated is stored in the data disk through the aggregator, and the data to be aggregated stored in the data disk is verified according to the md5 value. If the verification passes, the data block information list of the segmented file is modified.

7. The method according to claim 1, characterized in that The method further comprises: For each index node, when the file end reads data corresponding to the index node for the first time, a pre-read cache instance corresponding to the index node is created; Querying a preset number of segment files in the index node from the index disk to obtain segment file identifiers of the segment files and a data block information list of the segment files; Fill the segment file identifier of each segment file and the data block information list of each segment file into the pre-read cache instance corresponding to the index node; According to the pre-read cache instance corresponding to the index node, the data corresponding to the index node is read.

8. A data processing device, characterized in that: The device includes: The file storage module is used for, for a current data block to be stored, if the current data block is transferred from a file end, saving data block query information for querying the current data block into a data block information list in a first segmentation file, and recording a segmentation file identifier of the first segmentation file into an index node corresponding to a target file to which the first segmentation file belongs; the first segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the target data is divided, and the target file is a file set for the target data; The object storage module is used for storing a current data block, if the current data block is obtained by converting an object input from an object end, saving data block query information in the object metadata for querying the current data block into a data block information list in a second segmentation file according to the object metadata of the object, and recording the segmentation file identifier of the second segmentation file into an index node corresponding to the object; the second segmentation file is recorded in the index disk; the current data block is one of the data blocks into which the data corresponding to the object is divided.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images