A method for parsing metadata of a distributed storage system and extracting data

Abstract

The application discloses a kind of methods for resolving distributed storage system metadata and extracting data, characterized in that including the following steps: S100: obtaining the SSD flash data of distributed storage device;S200: obtaining the cluster, block structure of SSD flash of distributed storage device;S300: the data obtained and stored in buffer is converted to hexadecimal;S400: in unit of block, using identifier to find and obtain each boundary field of metadata in block;S500: parse metadata structure and obtain metadata;S600: verify the chunk and meta data obtained;S700: reorganize the metadata obtained and generate database chain.The method provided by the application can recover and extract data in the distributed storage device by resolving metadata when data is lost in the distributed storage device.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of analysis and data recovery of distributed storage systems, and relates to a method for analyzing metadata of a distributed storage system and extracting data, in particular, to a method for analyzing metadata of a distributed storage system and extracting data in a distributed storage device. BACKGROUND

[0002] In modern computing environments, distributed storage devices have become one of the key technologies for handling large-scale data, and have been widely used in distributed storage, clusters, and cloud platforms. Compared with traditional single-machine file systems (such as NTFS, EXT4, XFS, etc.), distributed storage systems have significantly different requirements and difficulties in metadata management, mainly reflected in:

[0003] 1. The metadata of traditional file systems usually contains basic information such as file name, size, permission, timestamp, etc. These metadata are stored and managed in a centralized manner. In NTFS, EXT4, XFS, and other file systems, the structure of metadata is relatively simple and fixed, and data and metadata access occurs on a single storage device. Although these file systems perform well in small-scale applications, they have obvious shortcomings in scalability and performance when handling large-scale data and distributed environments.

[0004] 2. The metadata structure of distributed storage systems is more complex. Its metadata not only manages the basic properties of files, but also handles the distribution of data on multiple storage nodes, the location of data blocks, the management of data replicas, the communication state between nodes, etc. In addition, the metadata in distributed storage systems is often dynamic, and as the system expands and data grows, the size and complexity of the metadata will increase dramatically.

[0005] Therefore, the application of traditional single-machine file systems to metadata cannot be used for distributed storage devices. The existing technology for analyzing the metadata of distributed storage devices has the following shortcomings:

[0006] 1. Efficiency of metadata analysis: In distributed storage devices, metadata is distributed across multiple nodes, and analyzing these metadata requires crossing multiple storage devices and multiple networks. Existing analysis tools are difficult to handle such complex distributed structures.

[0007] 2. Data consistency and reliability: In distributed storage devices, any update to metadata needs to ensure consistency among multiple copies. Existing technologies often have difficulty effectively ensuring data consistency when dealing with distributed metadata, especially in high-concurrency and high-failure-rate environments, making analysis even more difficult.

[0008] 3. Scalability: Unlike traditional file systems, distributed storage devices often need to handle large-scale data sets and frequent expansion needs. Existing metadata management methods often face performance bottlenecks when facing rapidly growing metadata, making it difficult to adapt to the expansion needs of the system and adding difficulty to the metadata analysis of the distributed storage device.

[0009] 4. Complex storage structure: The metadata structure in a distributed storage system is often composed of complex data structures that require special parsing methods to obtain useful information. Existing metadata analysis methods for distributed storage devices are difficult to apply in such complex storage structures.

[0010] The technical problems caused by the above shortcomings are: when the data in the distributed storage device is lost, it is difficult to recover and extract the data in the distributed storage device through the method of analyzing the metadata, and the resulting huge loss. SUMMARY

[0011] To overcome the shortcomings of the prior art, the present application provides a method for analyzing metadata of a distributed storage system and extracting data: by analyzing the structure of the metadata, the efficiency and accuracy of metadata analysis are effectively improved, ensuring the management of metadata and the security of data in a large-scale and highly complex distributed environment, and the data of the distributed storage device can be completely extracted and the lost data can be recovered, including the following steps:

[0012] S100: Obtain the SSD flash data of the distributed storage device;

[0013] S200: Obtain the cluster and block structure of the SSD flash of the distributed storage device, including the number of clusters contained in each block and the number of sectors contained in each cluster:

[0014] Take the first address of the partition table as the starting address, take 0xC5 as the offset, address and read the contents of the next 2 bytes as the number of sectors contained in the current cluster, where the offset is an integer starting from 0;

[0015] Take the first address of the partition table as the starting address, take 0xC7 as the offset, address and read the contents of the next 2 bytes as the number of clusters contained in the current block, where the offset is an integer starting from 0;

[0016] S300: Convert the data obtained and stored in the buffer to hexadecimal;

[0017] S400: In block unit, identifier is used to find and acquire each boundary field of metadata in the block, including chunk, meta chunk and meta data, wherein, chunk identifier is 0x6368756E6B, meta chunk identifier is 0x6D6574615F6368756E6B, and meta data identifier is 0x6D6574615F64617461.

[0018] S500: Analyzing the structure of metadata and acquiring metadata, including:

[0019] S501: Analyzing the structure of chunk and acquiring physical hard disk number and block number where the chunk is located, chunk number and block occupation identifier.

[0020] S502: Analyzing the structure of meta chunk and acquiring valid identifier, record number and byte length of meta data; at the same time, acquiring meta id, byte length of data between the first meta id and the last meta id, block number where the first meta id in the current block is located and interval information.

[0021] S503: Analyzing the structure of meta data and acquiring attribute information of the current meta data, including meta id corresponding to the current meta data, file name / directory name, byte length, end identifier and cluster number before the current chunk is occupied.

[0022] S600: Verifying the acquired chunk and meta data, including data length verification and format checking, to ensure the integrity and accuracy of the data.

[0023] S700: Reorganizing the acquired metadata and generating database chain, including the following steps:

[0024] S701: According to the structure of the analyzed meta chunk, acquiring complete meta data.

[0025] S702: Acquiring meta id of each metadata in meta data.

[0026] S703: Using meta id, splicing meta data and chunk and generating database chain, to extract metadata in SSD flash.

[0027] Preferably, the metadata is stored in little-endian format.

[0028] Preferably, step S501 includes the following steps:

[0029] S5011: search for a chunk identifier in the current block;

[0030] S5012: determine whether the chunk identifier is found in the current block, if yes, execute step S5014, otherwise, execute step S5013;

[0031] S5013: address the next block and execute step S5011;

[0032] S5014: address and read the content of the byte after the chunk identifier as the physical hard disk number where the current chunk is located;

[0033] address and read the content of the byte after the physical hard disk number as the byte length of the first data of the current chunk;

[0034] address and read the content of the byte after the byte length of the first data as the byte length of the second data of the current chunk;

[0035] S5015: address and read the content of the 0x10 bytes after the byte length of the second data as the first data of the current chunk, wherein the content of the first 8 bytes is the block number where the current chunk is located;

[0036] S5016: address and read the content of the 8 bytes after the first data of the chunk as the block occupancy identifier of the current chunk, wherein the block occupancy identifier of 0xFFFFFFFFFFFFFFFF represents that the data occupies the entire block;

[0037] S5017: address and read the content of the 4 bytes after the block occupancy identifier in step S5016 as the number of the current chunk.

[0038] Preferably, step S502 comprises the following steps:

[0039] S5021: search for a meta chunk identifier in the current block;

[0040] S5022: determine whether the meta chunk identifier is found in the current block, if yes, execute step S5023, otherwise, execute step S5021;

[0041] S5023: determine whether a valid identifier of the meta data is found in the current block, if yes, it indicates that the current metadata is valid data, execute step S5024, otherwise, continue to search and execute step S5021, wherein the valid identifier of the meta data is 0xCCCCCCCCCCCCCCCC;

[0042] S5024: Address and read the content of the next 8 bytes after the valid identifier, as the number of records contained in all meta data;

[0043] S5025: Address and read the content of the next 8 bytes after the number of records, as the byte length of all meta data;

[0044] S5026: Address and read the content of the next 8 bytes after the byte length of step S5025, as the first meta id, and read the content of the next 8 bytes after the first meta id, as the last meta id;

[0045] S5027: Address and read the content of the next 8 bytes after the last meta id, as the byte length of the data between the first meta id and the last meta id;

[0046] S5028: Address and read the content of the next 8 bytes after the byte length of step S5027, as the starting block number of the meta chunk storage;

[0047] S5029: Address and read the content of the next 8 bytes after the starting block number, as the interval information, wherein the interval information is 0x0000000000000000, indicating that the current meta chunk has not been recorded yet, and will continue to be recorded in the subsequent meta data.

[0048] Preferably, step S503 comprises the following steps:

[0049] S5031: Find the meta data identifier in the current block;

[0050] S5032: Determine whether the meta data identifier is found in the current block, if yes, execute step S5034, otherwise, execute step S5032;

[0051] S5032: Address the next block and execute step S5031;

[0052] S5034: Address and read the content of the next 1 byte after the meta data identifier, as the byte length of the current file name / directory name; address and read the content of the next 1 byte after the byte length of the current file name / directory name, as the byte length of the current chunk number, end identifier, number of clusters occupied before the current chunk, byte length of meta data, and meta id corresponding to the current meta data;

[0053] S5035: Address and read the content of 0x17 bytes after the common occupied byte length in step S5034 as the current file name / directory name;

[0054] S5036: Address and read the content of 0x18 bytes after the current file name / directory name, wherein the content of the first 4 bytes is the current chunk number; the content of the subsequent 4 bytes after the end identifier; the content of the 4 bytes after the end identifier is the number of clusters occupied by the current meta data before the current chunk; the content of the 4 bytes after the cluster number is the byte length of the current meta data; and the last 8 bytes of content is the meta id of the current meta data.

[0055] Preferably, the verification method of step S600 includes data length verification and format checking to ensure the integrity and accuracy of the data.

[0056] The beneficial effects of the present application are:

[0057] 1. Accuracy improvement: In the complex and variable storage requirements of distributed storage devices, the accuracy and speed of metadata analysis, especially the analysis of chunks and meta chunks.

[0058] 2. Adaptability and expansibility improvement: Compared with the prior art which can only be used in a single device or fixed structure and is difficult to cope with the complex scenarios of multiple nodes and multiple structures in a distributed storage system, the present application introduces metadata classification and optimization, which not only breaks through the limitations of traditional methods in structure, but also significantly improves the analysis efficiency and data management, making the present application have stronger adaptability and expansibility when processing large-scale data and distributed storage systems.

[0059] 3. Flexibility and security improvement: By subdividing and optimizing the management of metadata, the present application can more flexibly process and manage data in a distributed storage system, ensure efficient operation performance when the data volume increases or the system expands, better protect the integrity and security of data, and prevent the risk of data loss or tampering. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The overall flowchart of the present application;

[0061] Figure 2 The data structure diagram of the cluster and block structure in the embodiment of the present application;

[0062] Figure 3 The specific flowchart of analyzing the data structure of the chunk and obtaining the chunk information in the embodiment of the present application;

[0063] Figure 4 An example diagram of the data structure of a chunk in an embodiment of the present application is shown in FIG. 1.

[0064] Figure 5 An example diagram of the data structure of a meta chunk in an embodiment of the present application is shown in FIG. 2.

[0065] Figure 6 An example diagram of the data structure of a meta chunk in an embodiment of the present application is shown in FIG. 2.

[0066] Figure 7 An example diagram of the data structure of a meta chunk in an embodiment of the present application is shown in FIG. 2.

[0067] Figure 8 An example diagram of the data structure of a meta chunk in an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0068] The following terms may be used in the present application:

[0069] Metadata of a distributed system: refers to data that describes data, which is mainly used to describe and manage information of files and directories in a distributed system. Metadata contains various key attributes about data, such as file name, file size, creation time, access permission, storage location, etc.

[0070] Chunk: mainly used to record the physical location and distribution of data blocks, ensuring data consistency and reliability in a multi-node environment. Compared with the fixed location metadata structure in traditional systems, chunk provides a dynamic distribution solution, adapting to the high scalability requirements in distributed storage.

[0071] Meta chunk: used to manage the metadata of chunk, recording the association and storage relationship between multiple chunks. The lack of effective management of the relationship between data blocks in traditional file systems is solved by the introduction of meta chunk, ensuring efficient retrieval and recovery of data.

[0072] Meta data: similar to the metadata of traditional file systems, records the basic attributes of files (such as file name, size, permission, etc.), but on this basis, adds information related to distributed environments, such as the number of copies, storage node information, etc., significantly enhancing the functionality and flexibility of metadata.

[0073] meta id: As a unique identifier of metadata, it is used for quick positioning and accessing metadata in distributed storage. In traditional file systems, the positioning of metadata usually depends on the path or index, while the design of meta id is suitable for large-scale distributed systems, providing a more efficient access mechanism.

[0074] The application will be further described below in conjunction with the accompanying drawings and examples. Unless otherwise specified, the metadata in the application is stored in little-endian format.

[0075] Figure 1 The overall flowchart of the application is shown.

[0076] As Figure 1 shown, the method of the application comprises the following steps:

[0077] S100: The system metadata physical storage unit of the distributed storage device is usually an SSD flash, so first obtain the SSD flash data of the distributed storage device;

[0078] S200: Obtain the cluster and block structure of the SSD flash of the distributed storage device, including the number of clusters contained in each block and the number of sectors contained in each cluster:

[0079] Take the first address of the partition table as the starting address and 0xC5 as the offset, address and read the contents of the next 2 bytes as the number of sectors contained in the current cluster, wherein the offset is an integer starting from 0;

[0080] Take the first address of the partition table as the starting address and 0xC7 as the offset, address and read the contents of the next 2 bytes as the number of clusters contained in the current block, wherein the offset is an integer starting from 0;

[0081] Figure 2 The data structure example diagram of the cluster and block structure in the embodiment of the application is shown. As Figure 2 shown, take the first address of the partition table as the starting address and 0xC5 as the offset, address and read the contents of the next 2 bytes 0x0800 as the number of sectors contained in the current cluster, i.e., the current cluster contains 2048 sectors.

[0082] Take the first address of the partition table as the starting address and 0xC7 as the offset, address and read the contents of the next 2 bytes 0x0001 as the number of clusters contained in the current block, i.e., the current block contains 1 cluster.

[0083] S300: Convert the data obtained and stored in the buffer to hexadecimal;

[0084] S400: In block unit, the identifier is looked up to acquire each boundary field of metadata in the block, including chunk, meta chunk and meta data, wherein the chunk identifier is 0x6368756E6B, the meta chunk identifier is 0x6D6574615F6368756E6B, and the meta data identifier is 0x6D6574615F64617461.

[0085] According to different types of metadata, such as chunk, meta chunk, meta data and meta id, the data stream is classified and stored and labeled. Each type of data corresponds to different metadata categories according to the hexadecimal number identifier of the block header, which lays a foundation for subsequent in-depth analysis.

[0086] S500: The metadata structure is parsed to acquire metadata, including:

[0087] S501: The structure of the chunk is parsed to acquire the physical hard disk number and block number of the chunk, the number of the chunk and the block occupation identifier, and the occupation identifier indicates whether the current chunk completely occupies the entire block.

[0088] Figure 3 The specific flowchart of parsing the data structure of the chunk and acquiring the chunk information in the embodiment of the application is shown in FIG. 5. Figure 3 As shown in FIG. 5, step S501 includes the following steps:

[0089] S5011: The chunk identifier 0x6368756E6B in the current block is looked up.

[0090] S5012: It is judged whether the chunk identifier 0x6368756E6B is found in the current block, if yes, step S5014 is executed, otherwise, step S5013 is executed.

[0091] S5013: The next block is addressed, and step S5011 is executed.

[0092] S5014: The content of 1 byte after the chunk identifier is addressed and read as the physical hard disk number of the current chunk.

[0093] The content of 1 byte after the physical hard disk number is addressed and read as the byte length of the first data of the current chunk.

[0094] The content of 1 byte after the byte length of the first data is addressed and read as the byte length of the second data of the current chunk.

[0095] Figure 4A data structure example diagram of the chunk in the embodiment of the present application is shown.

[0096] As shown in Figure 4 , the content 0x04 of the 1 byte after the chunk identifier 0x6368756E6B is addressed and read as the physical hard disk number where the current chunk is located, i.e., the physical hard disk number where the current chunk is located is 4.

[0097] As shown in Figure 4 , the content 0x10 of the 1 byte after the physical hard disk number is addressed and read as the byte length of the first data of the current chunk; i.e., the byte length of the first data of the current chunk is 0x10.

[0098] As shown in Figure 4 , the content 0x04 of the 1 byte after the byte length of the first data is addressed and read as the byte length of the second data of the current chunk; i.e., the byte length of the second data of the current chunk is 0x10.

[0099] S5015: The content of the 0x10 bytes after the byte length of the second data is addressed and read as the first data of the current chunk, wherein the content of the first 8 bytes is the block number where the current chunk is located.

[0100] As shown in Figure 4 , the content 0xD22BCE0000000000 of the first 8 bytes, i.e., the block number where the current chunk is located is 0xD22BCE0000000000.

[0101] S5016: The content of the 8 bytes after the first data of the chunk is addressed and read as the block occupancy identifier of the current chunk, wherein the block occupancy identifier is 0xFFFFFFFFFFFFFFFF indicating that the data occupies the entire block.

[0102] As shown in Figure 4 , the content of the last 8 bytes is the block occupancy identifier, and its value is 0xFFFFFFFFFFFFFFFF, indicating that the current chunk occupies the entire block.

[0103] S5017: The content of the 4 bytes after the block occupancy identifier in step S5016 is addressed and read as the number of the current chunk.

[0104] As shown in Figure 4 , the content 0x54802E5E of the 4 bytes after the block occupancy identifier in step S5016 is addressed and read as the number of the current chunk.

[0105] At this point, the method of analyzing the data structure of a chunk and obtaining the chunk information has been described.

[0106] Similarly, it can be parsed Figure 4 The data structure of the next chunk is determined and chunk information is obtained.

[0107] For example, such as Figure 5 As shown, the chunk identifier 0x6368756E6B followed by 0x16 indicates that the metadata is stored on the physical hard drive numbered 0x16. 0x14 and 0x04 represent the byte lengths of the first and second data in the current chunk, respectively. Specifically, the first 8 bytes of the first data (0xD72BCE0000000000) represent the block number of the current chunk, the next 2 bytes (0x0008) indicate that the current chunk only uses the first 0x8 clusters within the current block, and the following 8 bytes... The first byte contains 0xDA2BCE0000000000, indicating the block number of the other block currently occupied by the chunk. The next two bytes, 0x00F8, indicate that the current chunk occupies the first 0xF8 clusters of block 0xDA2BCE0000000000. Here, 0x00 in 0x00F8 indicates that the chunk occupies the starting cluster of block 0xDA2BCE0000000000, while 0xF8 indicates that the chunk occupies the ending cluster of block 0xDA2BCE0000000000. In other words, the current chunk occupies 2 blocks.

[0108] S502: Parse the structure of the meta chunk and obtain the valid identifier, number of records, and byte length of the meta data; at the same time, obtain the meta id, the byte length of the data between the first meta id and the last meta id, the block number of the first meta id in the current block, and the interval information;

[0109] Figure 5 This is a flowchart illustrating the specific process of parsing the structure of a meta chunk and obtaining meta data information in an embodiment of the present invention. For example... Figure 6 As shown, step S502 includes the following steps:

[0110] S5021: Locate the meta chunk identifier 0x6D6574615F6368756E6B04 in the current block;

[0111] S5022: Determine whether the meta chunk identifier 0x6D6574615F6368756E6B04 is found in the current block. If yes, proceed to step S5023; otherwise, proceed to step S5021.

[0112] S5023: judging whether the valid identifier of the meta data is found in the current block, if yes, it means that the current meta data is valid data, and step S5024 is executed, otherwise, the searching is continued and step S5021 is executed, wherein the valid identifier of the meta data is 0xCCCCCCCCCCCCCCCC;

[0113] Figure 6 A data structure example diagram of the meta chunk in the embodiment of the application is shown.

[0114] As shown in Figure 6 , the identifier 0x6D6574615F6368756E6B04 represents that the block is a meta chunk, and only one meta chunk data exists in one block of the SSD flash memory, but the meta chunk occupies the whole block. Next, the byte length of each data segment in the meta chunk is 8 bytes. Among them, 0xCCCCCCCCCCCCCCCC represents that the meta data is valid, if it is not 0xCCCCCCCCCCCCCCCC, it means that the meta data is invalid, and the meta chunk needs to be searched again until 0xCCCCCCCCCCCCCCCC is found to be considered as a valid meta chunk.

[0115] S5024: addressing and reading the content of the 8 bytes after the valid identifier as the record number contained by all the meta data;

[0116] As shown in Figure 6 , the content of the 8 bytes after the valid identifier 0x4C4CBB3B04000000 represents that the record number contained by all the meta data is 0x4C4CBB3B04000000.

[0117] S5025: addressing and reading the content of the 8 bytes after the record number as the byte length of all the meta data;

[0118] As shown in Figure 6 , the content of the 8 bytes after the record number 0x0062790A4E000000 represents that the byte length of all the meta data is 0x0062790A4E000000 bytes.

[0119] S5026: addressing and reading the content of the 8 bytes after the byte length in step S5025 as the first meta id, and reading the content of the 8 bytes after the first meta id as the last meta id;

[0120] As shown in Figure 6As shown, the first meta ID is 0x0000000000000000, followed by 8 bytes of content 0x54802E5E00000000, and the last meta ID is 0x54802E5E00000000.

[0121] S5027: Address and read the content of the last 8 bytes after the last meta id, as the length of bytes occupied by the data between the first meta id and the last meta id;

[0122] like Figure 6 As shown, the content of the last 8 bytes after the last meta ID, 0x1C99D4AB08000000, represents the length of bytes occupied by the data between meta ID 0x0000000000000000 and the last meta ID 0x54802E5E00000000.

[0123] S5028: Address and read the contents of the next 8 bytes after the byte length in step S5027, and use them as the starting block number for storing the meta chunk;

[0124] like Figure 6 As shown, the 8 consecutive bytes following 0x1C99D4AB08000000 contain the value 0x0008000000000000, representing the starting block number of the meta chunk storage.

[0125] S5029: Address and read the contents of the first 8 bytes following the starting block number as interval information. The interval information is 0x0000000000000000, indicating that the current meta chunk has not been completely recorded and will continue to be recorded in subsequent metadata.

[0126] like Figure 7 As shown, the 8 consecutive bytes following the starting block number, 0x0000000000000000, serve as interval information, indicating that the current meta chunk has not yet been fully recorded and will continue to be recorded in subsequent metadata.

[0127] like Figure 7As shown, 0x55802E5E000000004C4CBB3B04000000 represents that the meta id is from 0x55802E5E00000000 to 0x4C4CBB3B0400000, 0xE4C8A45E4500000020D5000000000000 represents the byte length and position of the meta data storage, wherein 0xE4C8A45E45000000 represents the byte length of the current meta data, and 0x20D5000000000000 represents that the storage starts from the block of 0x20D5000000000000.

[0128] S503: parsing the structure of the meta data and obtaining attribute information of the current meta data, including the meta id corresponding to the current meta data, the file name / directory name, the byte length, the end identifier, the number of clusters occupied before the current chunk;

[0129] Figure 8 A specific flowchart of parsing the data structure of the meta data and obtaining the attribute information of the meta data in the embodiment of the application is shown.

[0130] As shown in Figure 8 S503 includes the following steps:

[0131] S5031: searching for the meta data identifier 0x6D6574615F64617461 in the current block;

[0132] S5032: judging whether the meta data identifier 0x6D6574615F64617461 is found in the current block, if yes, executing step S5034, otherwise, executing step S5032;

[0133] S5032: addressing the next block and executing step S5031;

[0134] Figure 8 A data structure example diagram of the meta data in the embodiment of the application is shown.

[0135] S5034: addressing and reading the content of the 1 byte after the meta data identifier, as the byte length of the current file name / directory name; addressing and reading the content of the 1 byte after the byte length of the current file name / directory name, as the byte length of the current chunk number, the end identifier, the number of clusters occupied before the current chunk, the byte length of the meta data and the meta id corresponding to the current meta data;

[0136] As shown inFigure 8 As shown, the content 0x17 of the 1 byte after the meta data identifier is addressed and read as the byte length of the current file name / directory name; the content 0x18 of the 1 byte after the byte length of the current file name / directory name is addressed and read as the byte length of the current chunk number, end identifier, the number of clusters occupied by the current chunk, the byte length of the meta data, and the meta id corresponding to the current meta data.

[0137] S5035: The content of the 0x17 bytes after the byte length commonly occupied in step S5034 is addressed and read as the current file name / directory name.

[0138] As shown, the content 0x32303138313030325F313032363636343533302E646F63 of the 0x17 bytes after the byte length commonly occupied in step S5034 is addressed and read as the file name of the current data. In this embodiment, 0x32303138313030325F313032363636343533302E646F63 is converted to ANSI code as 201810021026664530.doc, in other words, the file name is 20181002 1026664530.doc. ​

[0139] S5036: The content of the 0x18 bytes after the current file name / directory name is addressed and read, wherein the content of the first 4 bytes is the current chunk number; the content of the next 4 bytes is the end identifier; the content of the 4 bytes after the end identifier is the number of clusters occupied by the current meta data before the current chunk; the content of the 4 bytes after the cluster number is the byte length of the current meta data; and the last 8 bytes is the meta id of the current meta data.

[0140] As shown, the content 0x32303138313030325F313032363636343533302E646F63 of the 0x17 bytes after the byte length commonly occupied in step S5034 is addressed and read as the file name of the current data. In this embodiment, 0x32303138313030325F313032363636343533302E646F63 is converted to ANSI code as 201810021026664530.doc, in other words, the file name is 20181002 1026664530.doc. ​ ​As shown, the content of 0x18 bytes after the current file name is read and addressed, which is 0x54802E5E0000000010000000097E00004C4CBB3B04000000, wherein the content of the first 4 bytes is 0x54802E5E as the current chunk number; the content of the next 4 bytes is 0x00000000 as an end identifier, i.e., marking the end of the current chunk number, used to separate the chunk number and the next data; the content of the next 4 bytes is 0x10000000 after the end identifier, indicating the number of clusters occupied by the current metadata before the current chunk; the content of the next 4 bytes is 0x097e0000 after the cluster number, indicating the byte length of the current meta data; and the last 8 bytes indicate that the meta id of the current meta data is 0x4C4CBB3B04000000.

[0141] S600: verifying the obtained chunk and meta data, including data length verification and format checking, to ensure the integrity and accuracy of the data;

[0142] S700: recombining the obtained meta data and generating a database chain, including the following steps:

[0143] S701: obtaining the complete meta data according to the structure of the parsed meta chunk;

[0144] S702: obtaining the meta id of each meta data in the meta data;

[0145] S703: using the meta id to splice the meta data and the chunk and generate a database chain, to extract the meta data in the SSD flash memory.

[0146] In other words, the meta data contains the file name and the meta id, and the corresponding chunk data block storage location can be matched through the meta id.

[0147] For example, in the embodiments of the present application:

[0148] For the chunk, the parsed physical address, the start position of the data content, the block byte length, and the like can be stored in the corresponding data structure.

[0149] For the meta chunk, the parsed data can be mapped to the corresponding data structure to generate the physical position mapping of the data block and the relationship diagram between the meta data.

[0150] For the meta data, attribute information of the meta data file can be extracted, including the file name, the permission, the storage path and the like, combined with additional attributes specific to the distributed storage, such as the node position information, the copy state and the like.

[0151] For the meta id, the meta id can be converted into a unique identifier, used for quickly searching and accessing the meta data.

[0152] Through the method provided by the application, the complete database chain in the distributed storage, the cluster and the cloud platform can be extracted and recombined, and the technical problem that there is no method for parsing the meta data of the distributed storage system and extracting the data in the prior art is solved.

[0153] It should be understood that the application is not limited to the above examples, and those skilled in the art can improve or change the application according to the above description, and all the improvements and changes shall belong to the protection scope of the appended claims of the application.

Claims

1. A method for parsing metadata and extracting data from a distributed storage system, characterized in that... Includes the following steps: S100: Obtain SSD flash data from the distributed storage device; S200: Obtain the cluster and block structure of the SSD flash memory of the distributed storage device, including the number of clusters in each block and the number of sectors in each cluster: Starting from the first address of the partition table and using 0xC5 as the offset, address and read two consecutive bytes of content to determine the number of sectors contained in the current cluster, where the offset is an integer starting from 0. Starting from the first address of the partition table and using 0xC7 as the offset, address and read two consecutive bytes of content to determine the number of clusters contained in the current block, where the offset is an integer starting from 0; S300: Convert the acquired data stored in the buffer to hexadecimal; S400: Using identifiers, locate and retrieve the boundary fields of metadata in a block, including chunk, meta chunk, and meta data. The chunk identifier is 0x6368756E6B, the meta chunk identifier is 0x6D6574615F6368756E6B, and the meta data identifier is 0x6D6574615F64617461. S500: Parses the metadata structure and retrieves metadata, including: S501: Parse the structure of the chunk and obtain the physical hard disk number and block number where the chunk is located, the chunk number and the block occupancy identifier; S502: Parse the structure of the meta chunk and obtain the valid identifier, number of records, and byte length of the meta data; at the same time, obtain the meta id, the byte length of the data between the first meta id and the last meta id, the block number of the first meta id in the current block, and the interval information; S503: Parse the structure of the metadata and obtain the attribute information of the current metadata, including the metadata ID, file name / directory name, byte length, end marker, and number of clusters occupied before the current chunk; S600: Verify the acquired chunks and meta data, including data length verification and format checks, to ensure data integrity and accuracy; S700: Reorganize the acquired metadata and generate a database chain, including the following steps: S701: Obtain complete metadata based on the structure of the parsed meta chunk; S702: Retrieve the meta ID of each metadata element in the metadata; S703: Uses meta ID to concatenate meta data with chunks and generate a database chain to extract metadata from SSD flash memory.

2. The method for parsing metadata and extracting data from a distributed storage system according to claim 1, characterized in that, Metadata is stored in little-endian format.

3. The method for parsing metadata and extracting data from a distributed storage system according to claim 1, characterized in that, Step S501 includes the following steps: S5011: Find the chunk identifier in the current block; S5012: Determine whether a chunk identifier is found in the current block. If yes, proceed to step S5014; otherwise, proceed to step S5013. S5013: Address the next block, execute step S5011; S5014: Address and read the 1 byte following the chunk identifier, which is used as the physical hard drive number where the current chunk resides; Address and read the byte following the physical hard drive number, and use it as the length of the first data byte of the current chunk; Address and read the byte following the length of the first data, and use it as the length of the second data in the current chunk; S5015: Address and read the content of 0x10 consecutive bytes after the length of the second data, and use it as the first data of the current chunk. The first 8 bytes are used as the block number of the current chunk. S5016: Address and read the contents of the last 8 bytes of the first data in the chunk, which will be used as the block occupancy identifier of the current chunk. The block occupancy identifier of 0xFFFFFFFFFFFFFFFF indicates that the data occupies the entire block. S5017: Address and read the contents of the 4 consecutive bytes following the block occupancy identifier in step S5016, and use them as the number of the current chunk.

4. The method for parsing metadata and extracting data from a distributed storage system according to claim 1, characterized in that, Step S502 includes the following steps: S5021: Find the meta chunk identifier in the current block; S5022: Determine whether a meta chunk identifier is found in the current block. If yes, proceed to step S5023; otherwise, proceed to step S5021. S5023: Determine whether a valid identifier for meta data is found in the current block. If so, it means that the current metadata is valid data and proceed to step S5024. Otherwise, continue searching and proceed to step S5021. The valid identifier for meta data is 0xCCCCCCCCCCCCCCCC. S5024: Address and read the contents of the 8 consecutive bytes following the valid identifier, which will be used as the number of records contained in all metadata; S5025: Address and read the content of the next 8 consecutive bytes after the record number, which will be used as the byte length of all metadata; S5026: Address and read the content of the 8 consecutive bytes after the byte length described in step S5025 as the first meta id, and read the content of the 8 consecutive bytes after the first meta id as the last meta id; S5027: Address and read the contents of the last 8 bytes after the last meta ID, which will be used as the length of bytes occupied by the data between the first meta ID and the last meta ID; S5028: Address and read the contents of the 8 consecutive bytes following the byte length described in step S5027, and use them as the starting block number for storing the meta chunk; S5029: Address and read the contents of the first 8 bytes following the starting block number as interval information. The interval information is 0x0000000000000000, indicating that the current meta chunk has not been completely recorded and will continue to be recorded in subsequent metadata.

5. The method for parsing and extracting metadata from SSD flash memory according to claim 1, characterized in that, Step S503 includes the following steps: S5031: Locate the meta data identifier in the current block; S5032: Determine whether a meta data identifier is found in the current block. If yes, proceed to step S5034; otherwise, proceed to step S5032. S5032: Address the next block, execute step S5031; S5034: Address and read the 1 byte after the meta data identifier, as the byte length of the current file name / directory name; address and read the 1 byte after the byte length of the current file name / directory name, as the byte length occupied by the current chunk number, end marker, number of clusters before occupying the current chunk, byte length of the meta data, and the byte length occupied by the meta id corresponding to the current meta data. S5035: Address and read the contents of 0x17 consecutive bytes after the byte length occupied in step S5034, and use it as the current file name / directory name; S5036: Address and read the content of 0x18 bytes following the current filename / directory name. The first 4 bytes are the current chunk number; the next 4 bytes are the end marker; the next 4 bytes are the number of clusters occupied by the current meta data before the current chunk; the next 4 bytes are the byte length of the current meta data; and the last 8 bytes are the meta id of the current meta data.

6. The method for parsing metadata and extracting data from a distributed storage system according to claim 1, characterized in that, The verification method in step S600 includes data length verification and format checking to ensure the integrity and accuracy of the data.

Images