Method for analyzing metadata of distributed storage system and extracting data

Abstract

The invention discloses a method for analyzing metadata of a distributed storage system and extracting data. The method is characterized by comprising the following steps of S100, obtaining SSD flash memory data of a distributed storage device; s200, obtaining cluster and block structures of an SSD flash memory of the distributed storage device; s300, converting the acquired data which are stored in the buffer area into hexadecimal data; s400, by taking the block as a unit, searching and obtaining each boundary field of the metadata in the block by adopting the identifier; s500, analyzing the metadata structure and obtaining metadata; s600, the obtained chunk and the obtained meta data are verified; and S700, recombining the obtained metadata and generating a database chain. Through the method provided by the invention, when the data in the distributed storage device is lost, the data in the distributed storage device can be recovered and extracted through a metadata analysis method.

Description

Technical Field

[0001] The present invention belongs to the field of analysis and data recovery of distributed storage systems, and relates to a method for analyzing metadata of a distributed storage system and extracting data, and specifically, relates to a method for analyzing metadata of a distributed storage system and extracting data in a distributed storage device. Background Art

[0002] In modern computing environments, distributed storage devices have become one of the key technologies for processing large-scale data, and have been widely used in distributed storage, clusters, and cloud platforms. Compared with traditional stand-alone file systems (such as NTFS, EXT4, XFS, etc.), distributed storage systems have significantly different requirements and difficulties in metadata management, mainly reflected in:

[0003] 1. Metadata of traditional file systems usually includes basic information such as file name, size, permissions, timestamp, etc. These metadata are stored and managed in a centralized manner. In file systems such as NTFS, EXT4, and XFS, the structure of metadata is relatively simple and fixed, and the access of data and metadata occurs on a single storage device. Although these file systems perform well in small-scale applications, they have obvious deficiencies in scalability and performance in processing large-scale data and distributed environments.

[0004] 2. The metadata structure of a distributed storage system is more complex. Its metadata not only manages the basic attributes of files, but also needs to handle the distribution of data on multiple storage nodes, the location of data blocks, the management of data copies, the communication status between nodes, etc. In addition, the metadata in a distributed storage system is often dynamic. With the expansion of the system and the growth of data, the scale and complexity of metadata will increase dramatically.

[0005] Therefore, the application of metadata in traditional stand-alone file systems cannot be used in distributed storage devices. The existing technology for metadata analysis of distributed storage devices has the following deficiencies:

[0006] 1. Efficiency of metadata parsing: In distributed storage devices, metadata is distributed on multiple nodes. Parsing these metadata needs to span multiple storage devices and multiple networks. Existing parsing tools are difficult to handle this complex distributed structure.

[0007] 2. Data consistency and reliability: In distributed storage devices, any update of metadata needs to ensure consistency among multiple copies. Existing technologies often have difficulty effectively ensuring data consistency when processing distributed metadata, especially in environments with high concurrency and high failure rates, which makes analysis even more difficult.

[0008] 3. Scalability: Unlike traditional file systems, distributed storage devices usually need to handle large-scale data sets and frequent expansion requirements. Existing metadata management methods often produce performance bottlenecks when facing rapidly growing metadata, making it difficult to adapt to system expansion requirements and adding difficulties to metadata parsing of distributed storage devices.

[0009] 4. Complex storage structure: The metadata structure in a distributed storage system is often composed of complex data structures, which require special parsing methods to obtain useful information. However, the existing metadata parsing methods for distributed storage devices are difficult to apply to such complex storage structures.

[0010] The technical problem caused by the above-mentioned deficiencies is that when data in the distributed storage device is lost, it is difficult to recover and extract the data in the distributed storage device by parsing metadata, and the huge losses caused thereby. Summary of the invention

[0011] In view of the deficiencies in the prior art, the present application proposes a method for parsing metadata of a distributed storage system and extracting data: by parsing the structure of metadata, the efficiency and accuracy of metadata parsing are effectively improved, and metadata management and data security are guaranteed in a large-scale and highly complex distributed environment, and data of a distributed storage device can be completely extracted and lost data can be recovered, including the following steps:

[0012] S100: Acquire SSD flash memory data of a distributed storage device;

[0013] S200: Obtain the cluster and block structure of the SSD flash memory of the distributed storage device, including the number of clusters contained in each block and the number of sectors contained in each cluster:

[0014] Take the first address of the partition table as the starting address, 0xC5 as the offset, address and read the content of 2 consecutive bytes as the number of sectors contained in the current cluster, where the offset is an integer starting from 0;

[0015] Take the first address of the partition table as the starting address, 0xC7 as the offset, address and read the content of 2 consecutive bytes as the number of clusters contained in the current block, where the offset is an integer starting from 0;

[0016] S300: converting the data acquired and stored in the buffer into hexadecimal;

[0017] S400: using a block as a unit, using an identifier to search and obtain each boundary field of metadata in the block, including chunk, meta chunk, and meta data, wherein the chunk identifier is 0x6368756E6B, the meta chunk identifier is 0x6D6574615F6368756E6B, and the meta data identifier is 0x6D6574615F64617461;

[0018] S500: Parsing metadata structure and obtaining metadata, including:

[0019] S501: Parse the structure of the chunk and obtain the physical hard disk number and block number where the chunk is located, the chunk number and the block occupancy identifier;

[0020] S502: Parse the structure of the meta chunk and obtain the valid identifier, number of records and byte length of the meta data; at the same time, obtain the meta id, the byte length occupied by the data between the first meta id and the last meta id, the block number of the first meta id in the current block and the interval information;

[0021] S503: Parse the structure of the metadata and obtain the attribute information of the current metadata, including the metadata ID, file name / directory name, byte length, end mark, and number of clusters before the current chunk occupied by the current metadata.

[0022] S600: Verify the obtained chunk and meta data, including data length verification and format check, to ensure the integrity and accuracy of the data;

[0023] S700: Reorganizing the acquired metadata and generating a database chain, including the following steps:

[0024] S701: Obtain complete meta data according to the structure of the parsed meta chunk;

[0025] S702: Obtain the meta id of each metadata in the meta data;

[0026] S703: Using the meta id, the meta data is concatenated with the chunk and a database chain is generated to extract metadata from the SSD flash memory.

[0027] Preferably, the metadata is stored in little endian format.

[0028] Preferably, step S501 includes the following steps:

[0029] S5011: Find the chunk identifier in the current chunk;

[0030] S5012: Determine whether a chunk identifier is found in the current chunk. If so, execute step S5014; otherwise, execute step S5013.

[0031] S5013: address the next block and execute step S5011;

[0032] S5014: Address and read the content of the 1 byte after the chunk identifier as the physical hard disk number where the current chunk is located;

[0033] Address and read the content of the 1 byte after the physical hard disk number as the byte length of the first data of the current chunk;

[0034] Address and read the content of 1 byte after the byte length of the first data as the byte length of the second data of the current chunk;

[0035] S5015: addressing and reading the content of 0x10 bytes after the byte length of the second data as the first data of the current chunk, wherein the content of the first 8 bytes is used as the block number of the current chunk;

[0036] S5016: Address and read the 8 bytes after the first data of the chunk as the block occupancy identifier of the current chunk, wherein the block occupancy identifier is 0xFFFFFFFFFFFFFFFF indicating that the data occupies the entire chunk;

[0037] S5017: Address and read the 4 consecutive bytes of content after the block occupation identifier in step S5016 as the number of the current chunk.

[0038] Preferably, step S502 includes the following steps:

[0039] S5021: Find the meta chunk identifier in the current chunk;

[0040] S5022: Determine whether a meta chunk identifier is found in the current chunk. If so, execute step S5023; otherwise, execute step S5021.

[0041] S5023: Determine whether a valid identifier of the meta data is found in the current block. If so, it indicates that the current metadata is valid data, and step S5024 is executed. Otherwise, continue to search and execute step S5021, wherein the valid identifier of the meta data is 0xCCCCCCCCCCCCCCCC;

[0042] S5024: Address and read the 8-byte content after the valid identifier as the number of records contained in all meta data;

[0043] S5025: address and read the 8-byte content after the record number as the byte length of all meta data;

[0044] S5026: address and read the 8-byte content after the byte length in step S5025 as the first metaid, and read the 8-byte content after the first meta id as the last meta id;

[0045] S5027: address and read the 8-byte content after the last meta id as the byte length occupied by the data between the first meta id and the last meta id;

[0046] S5028: Address and read the 8-byte content after the byte length described in step S5027 as the starting block number for metachunk storage;

[0047] S5029: Address and read the 8 consecutive bytes after the starting block number as interval information, where the interval information is 0x0000000000000000, indicating that the current meta chunk has not been recorded completely and will continue to be recorded in subsequent metadata.

[0048] Preferably, step S503 includes the following steps:

[0049] S5031: Find the meta data identifier in the current block;

[0050] S5032: Determine whether a meta data identifier is found in the current block. If so, execute step S5034; otherwise, execute step S5032.

[0051] S5032: address the next block and execute step S5031;

[0052] S5034: address and read the content of the 1 byte after the meta data identifier as the byte length of the current file name / directory name; address and read the content of the 1 byte after the byte length of the current file name / directory name as the current chunk number, end mark, number of clusters before the current chunk, byte length of the meta data and the byte length occupied by the meta id corresponding to the current meta data;

[0053] S5035: Address and read the content of 0x17 bytes after the common byte length in step S5034 as the current file name / directory name;

[0054] S5036: Address and read the 0x18 bytes of content after the current file name / directory name, where the first 4 bytes are used as the current chunk number; the next 4 bytes are used as the end mark; the next 4 bytes after the end mark are used as the number of clusters before the current chunk occupied by the current meta data; the next 4 bytes after the number of clusters are used as the byte length of the current meta data; and the last 8 bytes are used as the meta id of the current meta data.

[0055] Preferably, the verification method of step S600 includes data length verification and format checking to ensure the integrity and accuracy of the data.

[0056] The beneficial effects of the present invention are:

[0057] 1. Improved accuracy: Under the complex and changing storage requirements of distributed storage devices, the accuracy and speed of metadata parsing, especially the parsing of chunks and meta chunks, are improved.

[0058] 2. Improved adaptability and scalability: Compared with the existing technology, which can only be performed under a single device or fixed structure and is difficult to cope with complex scenarios with multiple nodes and multiple structures in distributed storage systems, this application introduces the classification and optimization of metadata, which not only breaks through the limitations of traditional methods in structure, but also achieves significant improvements in parsing efficiency and data management, making this application more adaptable and scalable when processing large-scale data and distributed storage systems.

[0059] 3. Improved flexibility and security: Through the segmentation and optimized management of metadata, this application can more flexibly process and manage data in distributed storage systems, ensuring efficient operating performance when the amount of data increases or the system expands, and can better protect the integrity and security of data and prevent the risk of data loss or tampering. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is the overall flow chart of the present invention;

[0061] Figure 2 This is an example diagram of the data structure of the cluster and block structures in the embodiment of the present invention;

[0062] Figure 3 A specific flow chart of parsing a chunk data structure and obtaining chunk information in an embodiment of the present invention;

[0063] Figure 4 This is an example diagram of the data structure of a chunk in an embodiment of the present invention;

[0064] Figure 5 A specific flow chart of parsing the structure of a meta chunk and obtaining meta data information in an embodiment of the present invention;

[0065] Figure 6 An example diagram of the data structure of a meta chunk in an embodiment of the present invention;

[0066] Figure 7 The present invention is a specific flowchart of parsing the data structure of meta data and obtaining attribute information of meta data in an embodiment of the present invention.

[0067] Figure 8 An example diagram of the data structure of meta data in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The terms that may be used in the present invention are as follows:

[0069] Metadata of distributed systems: refers to data that describes data. It is mainly used to describe and manage information about files and directories in distributed systems. Metadata contains various key attributes about data, such as file name, file size, creation time, access rights, storage location, etc.

[0070] Chunk: Mainly used to record the physical location and distribution of data blocks to ensure data consistency and reliability in a multi-node environment. Compared with the fixed-location metadata structure in traditional systems, chunk provides a dynamic distribution solution to meet the high scalability requirements in distributed storage.

[0071] Meta chunk: Meta chunk is used to manage chunk metadata, recording the association and storage relationship between multiple chunks. Traditional file systems lack effective management of the relationship between data chunks, and the introduction of meta chunk solves this problem, ensuring efficient retrieval and recovery of data.

[0072] Meta data: Similar to the metadata of traditional file systems, it records the basic properties of files (such as file name, size, permissions, etc.), but adds information related to distributed environments, such as the number of replicas, storage node information, etc., which significantly enhances the functionality and flexibility of metadata.

[0073] Meta ID: As a unique identifier of metadata, it is used to quickly locate and access metadata in distributed storage. In traditional file systems, the location of metadata usually depends on paths or indexes, while the design of meta ID is suitable for large-scale distributed systems and provides a more efficient access mechanism.

[0074] The present invention is further described below in conjunction with the accompanying drawings and embodiments. Unless otherwise specified, metadata in the present invention application is stored in little-endian format.

[0075] Figure 1 An overall flow chart of the present invention is shown.

[0076] like Figure 1 As shown, the method of the present invention comprises the following steps:

[0077] S100: The physical storage unit of the system metadata of the distributed storage device is usually an SSD flash memory, so firstly, the SSD flash memory data of the distributed storage device is obtained;

[0078] S200: Obtain the cluster and block structure of the SSD flash memory of the distributed storage device, including the number of clusters contained in each block and the number of sectors contained in each cluster:

[0079] Take the first address of the partition table as the starting address, 0xC5 as the offset, address and read the content of 2 consecutive bytes as the number of sectors contained in the current cluster, where the offset is an integer starting from 0;

[0080] Take the first address of the partition table as the starting address, 0xC7 as the offset, address and read the content of 2 consecutive bytes as the number of clusters contained in the current block, where the offset is an integer starting from 0;

[0081] Figure 2 FIG. 2 shows an example of a data structure of a cluster and a block structure in an embodiment of the present invention. Figure 2 As shown, taking the first address of the partition table as the starting address and 0xC5 as the offset, addressing and reading the continuous 2-byte content 0x0800 as the number of sectors contained in the current cluster, that is, the current cluster contains 2048 sectors.

[0082] Taking the first address of the partition table as the starting address and 0xC7 as the offset, address and read the continuous 2-byte content 0x0001 as the number of clusters contained in the current block, that is, the current block contains 1 cluster.

[0083] S300: converting the data acquired and stored in the buffer into hexadecimal;

[0084] S400: using a block as a unit, using an identifier to search and obtain each boundary field of metadata in the block, including chunk, meta chunk, and meta data, wherein the chunk identifier is 0x6368756E6B, the meta chunk identifier is 0x6D6574615F6368756E6B, and the meta data identifier is 0x6D6574615F64617461;

[0085] According to different types of metadata, such as chunk, meta chunk, meta data, and meta id, the data stream is classified, stored, and labeled. Each type of data corresponds to a different metadata category based on the hexadecimal identifier of the block header, laying the foundation for subsequent in-depth analysis.

[0086] S500: Parsing metadata structure and obtaining metadata, including:

[0087] S501: Parse the structure of the chunk and obtain the physical hard disk number and block number of the chunk, the chunk number and the block occupancy identifier, where the occupancy identifier indicates whether the current chunk completely occupies the entire block;

[0088] Figure 3 FIG. 1 is a specific flow chart of parsing the data structure of a chunk and obtaining chunk information in an embodiment of the present invention. Figure 3 As shown, step S501 includes the following steps:

[0089] S5011: Find the chunk identifier 0x6368756E6B in the current chunk;

[0090] S5012: Determine whether the chunk identifier 0x6368756E6B is found in the current chunk. If so, execute step S5014; otherwise, execute step S5013.

[0091] S5013: address the next block and execute step S5011;

[0092] S5014: Address and read the content of the 1 byte after the chunk identifier as the physical hard disk number where the current chunk is located;

[0093] Address and read the content of the 1 byte after the physical hard disk number as the byte length of the first data of the current chunk;

[0094] Address and read the content of 1 byte after the byte length of the first data as the byte length of the second data of the current chunk;

[0095] Figure 4An example diagram of the data structure of a chunk in an embodiment of the present invention is shown.

[0096] like Figure 4 As shown, the content 0x04 of the 1 byte after the chunk identifier 0x6368756E6B is addressed and read as the physical hard disk number where the current chunk is located, that is, the physical hard disk number where the current chunk is located is 4.

[0097] like Figure 4 As shown, the content 0x10 of the 1 byte after the physical hard disk number is addressed and read as the byte length of the first data of the current chunk; that is, the byte length of the first data of the current chunk is 0x10.

[0098] like Figure 4 As shown, the content 0x04 of the byte after the byte length of the first data is addressed and read as the byte length of the second data of the current chunk; that is, the byte length of the second data of the current chunk is 0x10.

[0099] S5015: addressing and reading the content of 0x10 bytes after the byte length of the second data as the first data of the current chunk, wherein the content of the first 8 bytes is used as the block number of the current chunk;

[0100] like Figure 4 As shown, the content of the first 8 bytes is 0xD22BCE0000000000, that is, the block number of the current chunk is 0xD22BCE0000000000;

[0101] S5016: Address and read the 8 bytes after the first data of the chunk as the block occupancy identifier of the current chunk, wherein the block occupancy identifier is 0xFFFFFFFFFFFFFFFF indicating that the data occupies the entire chunk;

[0102] like Figure 4 As shown, the last 8 bytes are the block occupancy identifier, whose value is 0xFFFFFFFFFFFFFFFF, indicating that the current chunk occupies the entire block.

[0103] S5017: Address and read the 4 consecutive bytes of content after the block occupation identifier in step S5016 as the number of the current chunk.

[0104] like Figure 4 As shown, the 4 consecutive bytes of content 0x54802E5E after the block occupation identifier in step S5016 are addressed and read, which is the number of the current chunk.

[0105] So far, the method of parsing a chunk's data structure and obtaining chunk information has been described.

[0106] Similarly, it can be analyzed Figure 4 Get the data structure of the next chunk and obtain the chunk information.

[0107] For example, Figure 4 As shown in the figure, the 0x16 after the chunk identifier 0x6368756E6B of the next chunk indicates that the metadata is stored on the physical hard disk numbered 0x16, and 0x14 and 0x04 respectively indicate the byte length of the first data of the current chunk and the byte length of the second data; wherein, the first 8 bytes of the first data of the current chunk 0xD72BCE0000000000 represent the block number of the current chunk, and the next 2 bytes 0x0008 indicate that the current chunk only uses the first 0x8 clusters in the current block, and the next 8 bytes The section content is 0xDA2BCE0000000000, indicating the block number of another block occupied by the current chunk. The next 2 bytes of content 0x00F8 indicate that the current chunk occupies the first 0xF8 clusters of the 0xDA2BCE0000000000 block. Among them, 0x00 of 0x00F8 indicates that the chunk occupies the starting cluster of the 0xDA2BCE0000000000 block, and 0xF8 indicates that the chunk occupies the ending cluster of the 0xDA2BCE0000000000 block. In other words, the current chunk occupies 2 blocks.

[0108] S502: Parse the structure of the meta chunk and obtain the valid identifier, number of records and byte length of the meta data; at the same time, obtain the meta id, the byte length occupied by the data between the first meta id and the last meta id, the block number of the first meta id in the current block and the interval information;

[0109] Figure 5 FIG. 1 is a specific flow chart of parsing the structure of a meta chunk and obtaining meta data information in an embodiment of the present invention. Figure 5 As shown, step S502 includes the following steps:

[0110] S5021: Find the meta chunk identifier 0x6D6574615F6368756E6B04 in the current chunk;

[0111] S5022: Determine whether the meta chunk identifier 0x6D6574615F6368756E6B04 is found in the current chunk. If so, execute step S5023; otherwise, execute step S5021.

[0112] S5023: Determine whether a valid identifier of the meta data is found in the current block. If so, it indicates that the current metadata is valid data, and step S5024 is executed. Otherwise, continue to search and execute step S5021, wherein the valid identifier of the meta data is 0xCCCCCCCCCCCCCCCC;

[0113] Figure 6 An example diagram of the data structure of a meta chunk in an embodiment of the invention is shown.

[0114] like Figure 6 As shown in the figure, the identifier 0x6D6574615F6368756E6B04 indicates that the block is a meta chunk. There will only be one meta chunk data in an SSD flash memory, but the meta chunk will occupy the entire block. The byte length of each data segment in the metachunk is 8 bytes. Among them, 0xCCCCCCCCCCCCCCCC indicates that the metadata is valid. If it is not 0xCCCCCCCCCCCCCCCC, it means that the metadata is invalid, and the meta chunk needs to be searched again until 0xCCCCCCCCCCCCCCCCCC appears to be considered as a valid meta chunk.

[0115] S5024: Address and read the 8-byte content after the valid identifier as the number of records contained in all meta data;

[0116] like Figure 6 As shown, the 8 consecutive bytes of content 0x4C4CBB3B04000000 after the valid identifier indicate that the number of records contained in all metadata is 0x4C4CBB3B04000000.

[0117] S5025: address and read the 8-byte content after the record number as the byte length of all meta data;

[0118] like Figure 6 As shown, the 8 consecutive bytes after the record number are 0x0062790A4E000000, and the byte length of all meta data is 0x0062790A4E000000 bytes.

[0119] S5026: address and read the 8-byte content after the byte length in step S5025 as the first metaid, and read the 8-byte content after the first meta id as the last meta id;

[0120] like Figure 6As shown, the 8 consecutive bytes after the byte length 0x0000000000000000 are the first meta id, and the subsequent 8 consecutive bytes 0x54802E5E00000000 are the last meta id.

[0121] S5027: address and read the 8-byte content after the last meta id as the byte length occupied by the data between the first meta id and the last meta id;

[0122] like Figure 6 As shown, the 8 consecutive bytes of content 0x1C99D4AB08000000 after the last meta id represent the byte length occupied by the data between the meta id 0x0000000000000000 and the last meta id 0x54802E5E00000000.

[0123] S5028: Address and read the 8-byte content after the byte length in step S5027 as the starting block number for storing the meta chunk;

[0124] like Figure 6 As shown in the figure, the 8 consecutive bytes of content 0x00080000000000000 represent the starting block number of the meta chunk storage.

[0125] S5029: Address and read the 8 consecutive bytes after the starting block number as interval information, where the interval information is 0x0000000000000000, indicating that the current meta chunk has not been recorded completely and will continue to be recorded in subsequent metadata.

[0126] like Figure 6 As shown, the 8 consecutive bytes after the starting block number are 0x0000000000000000, which serves as interval information, indicating that the current meta chunk has not been recorded completely and will continue to be recorded in subsequent metadata.

[0127] like Figure 6As shown in the figure, 0x55802E5E000000004C4CBB3B04000000 indicates that the meta id is from 0x55802E5E00000000 to 0x4C4CBB3B0400000, and 0xE4C8A45E4500000020D50000000000000 indicates the byte length and position of the meta data storage. Among them, 0xE4C8A45E45000000 indicates the byte length of the current meta data, and 0x20D5000000000000 indicates that the storage starts from the block 0x20D5000000000000.

[0128] S503: Parse the structure of the metadata and obtain the attribute information of the current metadata, including the metadata ID, file name / directory name, byte length, end mark, and number of clusters before the current chunk occupied by the current metadata.

[0129] Figure 7 A specific flow chart of parsing the data structure of meta data and obtaining attribute information of meta data in an embodiment of the present invention is shown.

[0130] like Figure 7 As shown, step S503 includes the following steps:

[0131] S5031: Find the meta data identifier 0x6D6574615F64617461 in the current block;

[0132] S5032: Determine whether the meta data identifier 0x6D6574615F64617461 is found in the current block. If so, execute step S5034; otherwise, execute step S5032.

[0133] S5032: address the next block and execute step S5031;

[0134] Figure 8 An example diagram of the data structure of meta data in an embodiment of the invention is shown.

[0135] S5034: address and read the content of the 1 byte after the meta data identifier as the byte length of the current file name / directory name; address and read the content of the 1 byte after the byte length of the current file name / directory name as the current chunk number, end mark, number of clusters before the current chunk, byte length of the meta data and the byte length occupied by the meta id corresponding to the current meta data;

[0136] like Figure 8 As shown, address and read the content 0x17 of the 1 byte after the meta data identifier as the byte length of the current file name / directory name; address and read the content 0x18 of the 1 byte after the byte length of the current file name / directory name as the current chunk number, end mark, number of clusters before the current chunk, byte length of the meta data and the byte length occupied by the meta id corresponding to the current meta data;

[0137] S5035: Address and read the content of 0x17 bytes after the byte length occupied by the common file in step S5034 as the current file name / directory name;

[0138] like Figure 8 As shown, the content of 0x32303138313030325F313032363636343533302E646F63 of 0x17 bytes after the byte length occupied by the common in step S5034 is addressed and read as the file name of the current data. In this embodiment, 0x32303138313030325F313032363636343533302E646F63 is converted into ANSI code as 201810021026664530.doc, in other words, the file name is 20181002 1026664530.doc.

[0139] S5036: Address and read the 0x18 bytes of content after the current file name / directory name, where the first 4 bytes are used as the current chunk number; the next 4 bytes are used as the end mark; the next 4 bytes after the end mark are used as the number of clusters before the current chunk occupied by the current meta data; the next 4 bytes after the number of clusters are used as the byte length of the current meta data; and the last 8 bytes are used as the meta id of the current meta data.

[0140] like Figure 8As shown, the 0x18 bytes of content 0x54802E5E0000000010000000097E00004C4CBB3B04000000 are addressed and read after the current file name, where the first 4 bytes of content 0x54802E5E are used as the current chunk number; the following 4 bytes of content 0x00000000 are used as the end mark, that is, marking the end of the current chunk number to separate the chunk number from the next data; the 4 bytes of content 0x100000000 after the end mark indicate the number of clusters before the current chunk occupied by the current metadata; the 4 bytes of content 0x097e0000 after the number of clusters indicate the byte length of the current meta data; the last 8 bytes indicate that the meta id of the current meta data is 0x4C4CBB3B04000000.

[0141] S600: Verify the obtained chunk and meta data, including data length verification and format check, to ensure the integrity and accuracy of the data;

[0142] S700: Reorganizing the acquired metadata and generating a database chain, including the following steps:

[0143] S701: Obtain complete meta data according to the structure of the parsed meta chunk;

[0144] S702: Obtain the meta id of each metadata in the meta data;

[0145] S703: Using the meta id, the meta data is concatenated with the chunk and a database chain is generated to extract metadata from the SSD flash memory.

[0146] In other words, meta data contains the file name and meta id, and the meta id can be used to match the location where the corresponding chunk data block is stored.

[0147] For example, in the embodiments of the present invention:

[0148] For a chunk, the parsed physical address, the starting position of the data content, the block byte length and other information can be stored in a corresponding data structure.

[0149] For meta chunks, the parsed data can be mapped to the corresponding data structure to generate a physical location map of the data chunk, a relationship diagram between metadata, etc.

[0150] For meta data, the attribute information of the meta data file can be extracted, including file name, permissions, storage path, etc., combined with additional attributes unique to distributed storage, such as node location information, replica status, etc.

[0151] The meta id can be converted into a unique identifier for quick lookup and access to metadata.

[0152] Through the method provided by the present invention, a complete database chain in distributed storage, cluster, and cloud platform can be extracted and reorganized, solving the technical problem that there is no method for parsing metadata of distributed storage system and extracting data in the prior art.

[0153] It should be understood that the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for parsing metadata of a distributed storage system and extracting data, characterized in that The following steps are involved: S100: Acquire SSD flash memory data of a distributed storage device; S200: Obtain the cluster and block structure of the SSD flash memory of the distributed storage device, including the number of clusters contained in each block and the number of sectors contained in each cluster: Take the first address of the partition table as the starting address, 0xC5 as the offset, address and read the content of 2 consecutive bytes as the number of sectors contained in the current cluster, where the offset is an integer starting from 0; Take the first address of the partition table as the starting address, 0xC7 as the offset, address and read the content of 2 consecutive bytes as the number of clusters contained in the current block, where the offset is an integer starting from 0; S300: converting the data acquired and stored in the buffer into hexadecimal; S400: using a block as a unit, using an identifier to search and obtain each boundary field of metadata in the block, including chunk, meta chunk, and meta data, wherein the chunk identifier is 0x6368756E6B, the meta chunk identifier is 0x6D6574615F6368756E6B, and the meta data identifier is 0x6D6574615F64617461; S500: Parsing metadata structure and obtaining metadata, including: S501: Parse the structure of the chunk and obtain the physical hard disk number and block number where the chunk is located, the chunk number and the block occupancy identifier; S502: Parse the structure of the meta chunk and obtain the valid identifier, number of records and byte length of the meta data; at the same time, obtain the meta id, the byte length occupied by the data between the first meta id and the last meta id, the block number of the first meta id in the current block and the interval information; S503: Parse the structure of the meta data and obtain the attribute information of the current meta data, including the meta id, file name / directory name, byte length, end mark, and number of clusters before the current chunk corresponding to the current meta data; S600: Verify the obtained chunk and meta data, including data length verification and format check, to ensure the integrity and accuracy of the data; S700: Reorganizing the acquired metadata and generating a database chain, including the following steps: S701: Obtain complete meta data according to the structure of the parsed meta chunk; S702: Obtain the meta id of each metadata in the meta data; S703: Using the meta id, the meta data is concatenated with the chunk and a database chain is generated to extract metadata from the SSD flash memory.

2. A method for parsing metadata of a distributed storage system and extracting data according to claim 1, characterized in that: Metadata is stored in little endian format.

3. A method for parsing metadata of a distributed storage system and extracting data according to claim 1, characterized in that: Step S501 includes the following steps: S5011: Find the chunk identifier in the current chunk; S5012: Determine whether a chunk identifier is found in the current chunk. If so, execute step S5014; otherwise, execute step S5013. S5013: address the next block and execute step S5011; S5014: Address and read the content of the 1 byte after the chunk identifier as the physical hard disk number where the current chunk is located; Address and read the content of the 1 byte after the physical hard disk number as the byte length of the first data of the current chunk; Address and read the content of 1 byte after the byte length of the first data as the byte length of the second data of the current chunk; S5015: addressing and reading the content of 0x10 bytes after the byte length of the second data as the first data of the current chunk, wherein the content of the first 8 bytes is used as the block number of the current chunk; S5016: Address and read the 8 bytes after the first data of the chunk as the block occupancy identifier of the current chunk, wherein the block occupancy identifier is 0xFFFFFFFFFFFFFFFF indicating that the data occupies the entire chunk; S5017: Address and read the 4 consecutive bytes of content after the block occupation identifier in step S5016 as the number of the current chunk.

4. The method for parsing metadata of a distributed storage system and extracting data according to claim 1, characterized in that: Step S502 includes the following steps: S5021: Find the meta chunk identifier in the current chunk; S5022: Determine whether a meta chunk identifier is found in the current chunk. If so, execute step S5023; otherwise, execute step S5021. S5023: Determine whether a valid identifier of the meta data is found in the current block. If so, it indicates that the current metadata is valid data, and step S5024 is executed. Otherwise, continue to search and execute step S5021, wherein the valid identifier of the meta data is 0xCCCCCCCCCCCCCCCC; S5024: Address and read the 8-byte content after the valid identifier as the number of records contained in all meta data; S5025: address and read the 8-byte content after the record number as the byte length of all meta data; S5026: addressing and reading the 8-byte content after the byte length in step S5025 as the first meta id, and reading the 8-byte content after the first meta id as the last meta id; S5027: address and read the 8 consecutive bytes after the last metaid as the byte length occupied by the data between the first meta id and the last meta id; S5028: Address and read the 8-byte content after the byte length described in step S5027 as the starting block number for storing the meta chunk; S5029: Address and read the 8 consecutive bytes after the starting block number as interval information, where the interval information is 0x0000000000000000, indicating that the current meta chunk has not been recorded completely and will continue to be recorded in subsequent metadata.

5. The method for parsing and extracting metadata from SSD flash memory according to claim 1, characterized in that: Step S503 includes the following steps: S5031: Find the meta data identifier in the current block; S5032: Determine whether a meta data identifier is found in the current block. If so, execute step S5034; otherwise, execute step S5032. S5032: address the next block and execute step S5031; S5034: address and read the content of the 1 byte after the meta data identifier as the byte length of the current file name / directory name; address and read the content of the 1 byte after the byte length of the current file name / directory name as the current chunk number, end mark, number of clusters before the current chunk, byte length of the meta data and the byte length occupied by the meta id corresponding to the current meta data; S5035: Address and read the content of 0x17 bytes after the common byte length in step S5034 as the current file name / directory name; S5036: Address and read the 0x18 bytes of content after the current file name / directory name, where the first 4 bytes are used as the current chunk number; the next 4 bytes are used as the end mark; the next 4 bytes after the end mark are used as the number of clusters before the current chunk occupied by the current meta data; the next 4 bytes after the number of clusters are used as the byte length of the current meta data; and the last 8 bytes are used as the meta id of the current meta data.

6. A method for parsing metadata of a distributed storage system and extracting data according to claim 1, characterized in that: The verification method of step S600 includes data length verification and format check to ensure the integrity and accuracy of the data.

Images