Data storage method, device and equipment

Abstract

The invention provides a data storage method, device and equipment, and the method comprises the steps: obtaining a data storage request which comprises directory information; determining a target index structure based on the directory information, wherein the target index structure comprises multiple levels of index nodes corresponding to multiple levels of directories in the directory information; for each index node, if the index node is not the last-level index node of the multi-level index nodes, the metadata of the index node comprises the storage position information of the next-level index node of the index node, and if the index node is the last-level index node of the multi-level index nodes, the metadata of the index node comprises the storage position information of the next-level index node of the index node. The metadata of the index node comprises storage position information of to-be-stored data; for each index node, storing the metadata of the index node based on the storage position information of the index node; and storing the to-be-stored data based on the storage position information of the to-be-stored data. Through the scheme of the invention, the intercommunication between the object storage protocol and the distributed file storage protocol is realized.

Description

Data storage method, device and equipment TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a data storage method, device and equipment. BACKGROUND

[0002] Unstructured data is data whose structure is irregular or incomplete, has no predefined data model, and cannot be stored using a database logical table. Unstructured data includes social media content, images, audio, video, text, HTML (Hyper Text Markup Language), various reports, and the like. In order to store unstructured data, object storage and distributed file storage can be used.

[0003] Object storage is a method of storing data and metadata in association with objects as the basic storage unit. Object storage can flexibly store unstructured data, and has high scalability and reliability. For example, the S3 (Simple Storage Service) protocol is an object storage service, and the S3 protocol defines the access and management method of object storage, and can store unstructured data.

[0004] Distributed file storage is a method of storing unstructured data on multiple nodes. For example, the NFS (Network File System) protocol and the CIFS (Common Internet File System) protocol are network file system protocols for distributed file storage, and are used to share and access file data in a distributed environment. The NFS allows remote systems to access remote files in the manner of a local file system, and provides transparent file access capabilities. The CIFS allows devices of different operating systems to share files and printers, and provides powerful permission control, encryption and authentication functions.

[0005] SUMMARY

[0006] The present application provides a data storage method, which comprises:

[0007] Obtaining a data storage request, wherein the data storage request comprises directory information; if the data storage request is based on an object storage protocol, the directory information comprises a uniform resource locator; if the data storage request is based on a distributed file storage protocol, the directory information comprises a file path;

[0008] determine a target index structure based on the directory information, the target index structure comprising a plurality of index nodes corresponding to a plurality of levels of directories in the directory information; wherein for each index node, if the index node is not a last level index node of the plurality of index nodes, metadata of the index node comprises storage location information of a next level index node of the index node, and if the index node is the last level index node of the plurality of index nodes, the metadata of the index node comprises storage location information of to-be-stored data;

[0009] store, for each index node, the metadata of the index node based on the storage location information of the index node, and store the to-be-stored data based on the storage location information of the to-be-stored data.

[0010] The application provides a data storage apparatus, which comprises:

[0011] an obtaining module configured to obtain a data storage request, the data storage request comprising directory information; wherein if the data storage request is based on an object storage protocol, the directory information comprises a uniform resource locator, and if the data storage request is based on a distributed file storage protocol, the directory information comprises a file path;

[0012] a determining module configured to determine a target index structure based on the directory information, the target index structure comprising a plurality of index nodes corresponding to a plurality of levels of directories in the directory information; wherein for each index node, if the index node is not a last level index node of the plurality of index nodes, metadata of the index node comprises storage location information of a next level index node of the index node, and if the index node is the last level index node of the plurality of index nodes, the metadata of the index node comprises storage location information of to-be-stored data;

[0013] a storing module configured to store, for each index node, the metadata of the index node based on the storage location information of the index node, and store the to-be-stored data based on the storage location information of the to-be-stored data.

[0014] The application provides an electronic device, which comprises a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by the processor; the processor is configured to execute the machine executable instructions to implement the data storage method of the above examples.

[0015] The application provides a computer program product, which comprises a computer program, the computer program being executed by a processor to implement the data storage method of the above examples.

[0016] The application provides a machine readable storage medium storing machine executable instructions capable of being executed by a processor; wherein the processor is configured to execute the machine executable instructions to implement the data storage method of the above examples.

[0017] As can be seen from the above technical solutions, in the embodiments of the application, the intercommunication of the object storage protocol and the distributed file storage protocol can be implemented, the object storage protocol and the distributed file storage protocol can be supported to store data, the data stored by the distributed file storage protocol can be accessed by the object storage protocol, and the data stored by the object storage protocol can be accessed by the distributed file storage protocol. When different protocols are accessed, semantic loss and performance decay can be reduced, and space utilization can be maximized. Data can be intercommunicated between different protocols, so that the same data can be seen by a user when the user accesses the data using different protocols. This data intercommunication simplifies data management and data sharing and reduces the problem of data islands. Seamless data access and sharing enable the user to switch between different protocols without worrying about the inconsistency of data, and the change of data can be reflected in real time on the access path of all protocols. Since multiple independent storage systems for different protocols do not need to be maintained, the hardware, software and operation and maintenance costs can be significantly reduced, and the storage resources can be more efficiently managed by an administrator. BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1 is a flowchart of a data storage method according to an embodiment of the application;

[0019] FIG. 2 is a schematic diagram of an application scenario of artificial intelligence according to an embodiment of the application;

[0020] FIG. 3 is a schematic diagram of an architecture of a data storage system according to an embodiment of the application;

[0021] FIG. 4 is a schematic diagram of determining an index structure based on a multi-level directory according to an embodiment of the application;

[0022] FIG. 5 is a schematic diagram of a target version identifier according to an embodiment of the application;

[0023] FIG. 6 is a schematic diagram of a structure of a data storage device according to an embodiment of the application;

[0024] FIG. 7 is a hardware structure diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0025] A data storage method is provided in the embodiments of the application, which can be applied to an electronic device. The electronic device can be any type of device. Referring to FIG. 1, a flowchart of the method is shown. The method comprises the following steps.

[0026] In step 101, a data storage request is obtained, the data storage request comprising directory information. If the data storage request is based on an object storage protocol, the directory information comprises a uniform resource locator; if the data storage request is based on a distributed file storage protocol, the directory information comprises a file path.

[0027] In step 102, a target index structure is determined based on the directory information, the target index structure comprising a plurality of index nodes corresponding to a plurality of directories in the directory information. For each index node, if the index node is not a last index node of the plurality of index nodes, metadata of the index node comprises storage location information of a next index node of the index node; if the index node is the last index node of the plurality of index nodes, the metadata of the index node comprises storage location information of to-be-stored data.

[0028] In step 103, for each index node, metadata of the index node is stored based on the storage location information of the index node (e.g., the metadata is stored to a disk location corresponding to the storage location information); and to-be-stored data is stored based on the storage location information of the to-be-stored data (e.g., the to-be-stored data is stored to a disk location corresponding to the storage location information).

[0029] In one example, if the directory information comprises a file path, the file path is split into a plurality of directories based on a first separator in the file path; or, if the directory information comprises a uniform resource locator, the uniform resource locator is split into a plurality of directories based on a second separator in the uniform resource locator; wherein each character after a second separator in the uniform resource locator corresponds to a directory, and the second separator is / .

[0030] In one example, if a uniform resource locator in a storage request of an object storage protocol and a file path in a storage request of a distributed file storage protocol correspond to a same plurality of directories, the uniform resource locator and the file path correspond to a same target index structure; wherein the to-be-stored data is unstructured data; the object storage protocol comprises a simple storage service protocol; and the distributed file storage protocol comprises at least one of a network file system protocol, a common internet file system protocol, and a portable operating system interface protocol.

[0031] In one example, after storing the data to be stored based on the storage location information of the data to be stored, a data read request can be obtained, which can include directory information. If the data read request is based on an object storage protocol, the directory information can include a uniform resource locator. If the data read request is based on a distributed file storage protocol, the directory information can include a file path. A plurality of index nodes corresponding to a plurality of levels of directories in the directory information are determined. For each index node, metadata of the index node is read based on the storage location information of the index node. If the index node is not a last level index node of the plurality of index nodes, storage location information of a next level index node of the index node can be determined based on the metadata of the index node. If the index node is the last level index node of the plurality of index nodes, storage location information of stored data can be determined based on the metadata of the index node, and the stored data is read based on the storage location information.

[0032] In one example, if an interface creation request based on a distributed file storage protocol is obtained, a new index node is added in the target index structure based on the interface creation request. If an interface deletion request based on the distributed file storage protocol is obtained, an existing index node is deleted from the target index structure based on the interface deletion request. If a hard link interface request based on the distributed file storage protocol is obtained, a new index node is added in the target index structure, and the new index node points to the existing index node. If a soft link interface request based on the distributed file storage protocol is obtained, a new index node is added in the target index structure, and metadata of the new index node includes a string pointing to an arbitrary path. If an interface write request based on the distributed file storage protocol is obtained, data is written to an offset address specified by an index node of the target index structure based on the interface write request. If an interface read request based on the distributed file storage protocol is obtained, data is read from an offset address specified by an index node of the target index structure based on the interface read request.

[0033] In one example, if an interface read request based on an object storage protocol is obtained, a multi-level index node is determined based on the interface read request; for each index node, metadata of the index node is obtained; if the index node is a last level index node of the multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a hard link file, storage location information of stored data is determined based on the metadata of the index node, and the stored data is read based on the storage location information; or, if the index node is a last level index node of the multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a soft link file, path information of the soft link file is determined based on the metadata of the index node, a new index node is queried according to the path information of the soft link file, storage location information of stored data is determined based on metadata of the new index node, and the stored data is read based on the storage location information.

[0034] In one example, if a list object request based on an object storage protocol is obtained, the list object request is used to list all objects in a storage bucket, or the list object request is used to filter or group all objects in the storage bucket according to a prefix and a separator, a multi-level index node corresponding to the storage bucket is determined based on the list object request; for each index node, metadata of the index node is obtained; and metadata of the multi-level index node corresponding to the storage bucket is returned to the client in a specified order. In the process of storing the metadata of the multi-level index node, the metadata of the multi-level index node is stored in the specified order; in the process of obtaining the metadata of the multi-level index node corresponding to the storage bucket, the metadata of the multi-level index node corresponding to the storage bucket is obtained in the specified order; and the specified order includes a lexicographical order, and the lexicographical order is arranged in alphabetical order.

[0035] In one example, if a segmented upload request based on an object storage protocol is obtained, a multi-level index node is determined based on the segmented upload request; for each index node, metadata of the index node is obtained; if the index node is a last level index node of the multi-level index node, storage location information of to-be-stored data is determined based on the metadata of the index node; and data blocks in the segmented upload request are stored based on the storage location information of the to-be-stored data; wherein the to-be-stored data includes a plurality of data segments, and each data segment includes at least one data block; after the data blocks in the segmented upload request are stored, extension data is added in the metadata of the index node, and the extension data includes a segment identifier of a data segment to which the data block belongs, an overlay identifier of the data segment, a block identifier of the data block, and a size of the data block; and the overlay identifier of the data segment indicates a number of repetitions of the data segment.

[0036] In one example, if a read request based on a distributed file storage protocol or an object storage protocol get request is obtained, a multi-level index node is determined based on the read request or the get request; for each index node, metadata of the index node is obtained; if the index node is the last level index node of the multi-level index node, a target shard identifier of a data shard to be read is obtained based on the read request or the get request; a maximum coverage identifier corresponding to the target shard identifier is determined based on the metadata of the index node, target extension data corresponding to the target shard identifier and the maximum coverage identifier is determined from the metadata of the index node; based on a sorting position of the target extension data and a size of a data block in the target extension data, an offset address and a length of the data shard to be read are determined.

[0037] Based on the metadata of the index node, storage location information corresponding to the data shard to be read is determined, and the data shard to be read is obtained based on the storage location information, the offset address and the length.

[0038] In one example, if a truncate operation request based on a distributed file storage protocol is obtained, a multi-level index node is determined based on the truncate operation request; for each index node, metadata of the index node is obtained; if the index node is the last level index node of the multi-level index node, extension data of a plurality of data shards can be determined from the metadata of the index node, and the plurality of data shards are sorted based on the extension data; based on a sorting result and a truncate size in the truncate operation request, a data shard to be truncated is determined.

[0039] Based on a sorting position of the extension data of the data shard to be truncated and a size of each data block in the data shard to be truncated, an offset address and a length of the data shard to be truncated are determined.

[0040] Based on the metadata of the index node, storage location information corresponding to the data shard to be truncated is determined, and a clear operation is performed on the data shard to be truncated based on the storage location information, the offset address and the length.

[0041] In one example, for each inode, when generating the first metadata of the current version for the inode, a latest version identifier can be generated for the first metadata, and the latest version identifier can be added to the first metadata. If the inode has corresponded to the second metadata of the historical version, a snapshot identifier can be generated for the second metadata, and the snapshot identifier can be a negative value, and the snapshot identifier can be added to the second metadata. A target version identifier can be generated for the second metadata, and the target version identifier can be added to the second metadata. The target version identifier can include a first random character, a target ordered character, and a second random character. The target ordered character is obtained by performing an encryption operation on an initial ordered character and the first random character. The initial ordered character is obtained by performing an operation on the initial ordered character of the metadata of the previous version.

[0042] As can be seen from the above technical solutions, in the embodiments of the present application, the intercommunication of the object storage protocol and the distributed file storage protocol can be implemented, the object storage protocol and the distributed file storage protocol can be supported to store data, the object storage protocol can be supported to access the data stored by the distributed file storage protocol, and the distributed file storage protocol can be supported to access the data stored by the object storage protocol. When different protocols are accessed, semantic loss and performance decay can be reduced, and space utilization can be maximized. Data can be intercommunicated between different protocols, so that the same data can be seen by users when the data is accessed by using different protocols. This data intercommunication simplifies data management and data sharing, and reduces the problem of data islands. Seamless data access and sharing make users not need to worry about the inconsistency of data when switching between different protocols, and the change of data can be reflected in real time on the access path of all protocols. Since multiple independent storage systems need not be maintained for different protocols, the hardware, software, and operation and maintenance costs can be significantly reduced, and the storage resources can be more efficiently managed by administrators.

[0043] The data storage method of the embodiments of the present application will be described below in combination with specific application scenarios.

[0044] In order to implement the storage of unstructured data, object storage and distributed file storage can be used. For example, the object storage can include the S3 protocol, and the distributed file storage can include the NFS protocol, the CIFS protocol, and the POSIX (Portable Operating System Interface) protocol.

[0045] In order to store unstructured data, a plurality of independent storage systems need to be maintained. For example, a storage system 1 is maintained for an S3 protocol, a storage system 2 is maintained for an NFS protocol, and so on. For a storage request or a read request for the S3 protocol, data storage or data reading is implemented based on the storage system 1. For a storage request or a data read request for the NFS protocol, data storage or data reading is implemented based on the storage system 2.

[0046] In one example, artificial intelligence relies on unstructured data for training and reasoning to achieve understanding and analysis of information such as text, images, audio, and the like. Referring to FIG. 2, an application scenario of artificial intelligence is shown. Artificial intelligence involves a data collection process, a data preparation process, a model training process, and a reasoning process.

[0047] In the data collection process, raw data needs to be collected from a large number of devices, involving the collection of raw data using a plurality of protocols. For example, raw data is collected using the S3 protocol and stored in an S3 protocol storage pool, and raw data is collected using the NFS protocol and stored in an NFS protocol storage pool.

[0048] In the data preparation process, raw data needs to be read from the storage pool and processed, such as annotation processing, desensitization processing, anonymization processing, and cleaning processing. Therefore, a plurality of protocols need to be supported to implement the data preparation process. For example, raw data is read from the S3 protocol storage pool using the S3 protocol, processed, and stored in the S3 protocol storage pool. Raw data is read from the NFS protocol storage pool using the NFS protocol, processed, and stored in the NFS protocol storage pool. If a certain protocol (such as the S3 protocol) is not supported, raw data cannot be read from the S3 protocol storage pool, and raw data cannot be processed.

[0049] In the model training process, data needs to be read from the storage pool to train a model based on the data. Therefore, a plurality of protocols need to be supported to implement the model training process. For example, data is read from the S3 protocol storage pool using the S3 protocol, data is read from the NFS protocol storage pool using the NFS protocol, and a model is trained based on the read data. If a certain protocol (such as the S3 protocol) is not supported, data cannot be read from the S3 protocol storage pool, and the model cannot be trained based on the data.

[0050] In the reasoning process, the trained model can be deployed to a business cloud or a terminal, and the application can read the model of the business cloud or the terminal to perform reasoning on the input data and obtain a reasoning result.

[0051] For the above finding, this embodiment proposes a data storage method, which can realize the intercommunication of multiple protocols. For example, in the data collection process, the original data is collected by using the S3 protocol and stored in a general storage pool, and the original data is collected by using the NFS protocol and stored in the general storage pool, that is, different protocols correspond to the same general storage pool. In the data preparation process, only one kind of protocol can realize the data preparation process. For example, the original data is read from the general storage pool by using the S3 protocol (or the NFS protocol), the original data is processed, and the processed data is stored in the general storage pool. In the model training process, only one kind of protocol can realize the model training process. For example, the data is read from the general storage pool by using the S3 protocol (or the NFS protocol), and the model is trained based on the data.

[0052] In summary, the storage scheme of multiple protocol fusion can meet the storage and access requirements of different forms of data. For example, the object storage protocol (such as S3) and the distributed file storage protocol (such as NFS, CIFS and POSIX) can be fused, thereby providing an efficient and reliable data support environment for artificial intelligence, supporting artificial intelligence to train using unstructured data, and then implementing inference based on the trained model.

[0053] In this embodiment, a data storage system is proposed. Referring to FIG. 3, it is a schematic diagram of the architecture of the data storage system. The data storage system can include a protocol layer, an engine layer and a persistence layer.

[0054] The protocol layer is used to interface various storage protocols, responsible for accepting and parsing the requests (such as data storage requests or data reading requests) of various storage protocols sent by the client, determining the abstract semantics (i.e., multi-level directories) that the engine layer can recognize based on the request, and sending the multi-level directories to the engine layer.

[0055] In one example, the protocol layer supports an object storage protocol parsing module, which is used to accept and parse the requests of the object storage protocol sent by the client, and determine the multi-level directories that the engine layer can recognize based on the request. The object storage protocol can include but is not limited to the simple storage service protocol (S3 protocol), and the parsing module determines the multi-level directories based on the request after receiving the request of the S3 protocol.

[0056] For example, the request of the S3 protocol can include directory information, and the directory information can be a uniform resource locator (URL). Based on this, the uniform resource locator can be divided into multi-level directories based on the second separator in the uniform resource locator, that is, the characters after each second separator in the uniform resource locator correspond to a directory, and the second separator can be “ / ”.

[0057] For example, the semantics of the S3 protocol request does not have a path concept, but the separator in the S3 protocol request usually uses " / ", and the separator " / " in the S3 protocol request is consistent with the separator of the file path. Based on this, the uniform resource locator in the S3 protocol request can be separated by the separator " / ", and then the uniform resource locator is converted into a file path without loss. In this way, the bucket of the S3 protocol corresponds to a namespace, and then each separator " / " is identified as a directory, that is, the characters after each separator " / " in the uniform resource locator correspond to a directory, so that the S3 protocol request can be disassembled into multiple index nodes, compatible with the path semantics of the file system.

[0058] For example, based on the uniform resource locator in the S3 protocol request, the first separator " / " is followed by the first-level directory, the second separator " / " is followed by the second-level directory, the third separator " / " is followed by the third-level directory, and so on. The uniform resource locator can be divided into multiple directories, and the directory name of each directory can be obtained, and the directory name of each directory can be sent to the engine layer.

[0059] In one example, the protocol layer supports a distributed file storage protocol analysis module, which is used to accept and analyze the distributed file storage protocol request sent by the client, and determine the multi-level directory that can be recognized by the engine layer based on the request. Wherein, the distributed file storage protocol can include but not limited to network file system protocol (NFS protocol), common internet file system protocol (CIFS protocol), portable operating system interface protocol (POSIX protocol), these distributed file storage protocols can correspond to the same analysis module, or can correspond to different analysis modules. After receiving the request of the NFS protocol (or CIFS protocol, or POSIX protocol), the analysis module determines the multi-level directory based on the request.

[0060] For example, the request of the NFS protocol can include directory information, and the directory information can be a file path. Based on this, the file path can be divided into multiple directories based on the first separator in the file path, that is, the characters after each first separator in the file path correspond to a directory.

[0061] For example, the semantics of the NFS protocol request has a path concept, that is, the NFS protocol (or CIFS protocol, or POSIX protocol) is file semantics, which can uniquely identify a file through a file path, and the file path has explicitly indicated the directory, so the file path can be divided into multiple directories.

[0062] For example, the first first separator " / " is followed by a first level directory, the second first separator " / " is followed by a second level directory, and so on, so that the file path can be divided into multiple directories, and the directory name of each directory can be obtained, and the directory name of each directory can be sent to the engine layer.

[0063] The engine layer is used to process the abstract semantics (multi-level directory) and special semantics of all protocols, so that each protocol can make a request according to the native semantics and obtain a reply without losing semantics. The engine layer also manages metadata and data. For example, after receiving the multi-level directory in the directory information, the engine layer determines an index structure based on the multi-level directory, which can include a multi-level index node corresponding to the multi-level directory.

[0064] In one example, the engine layer can determine an index structure based on the multi-level directory, and store metadata and data in the form of the index structure. Referring to FIG. 4, an example of determining an index structure based on a multi-level directory is shown. In FIG. 4, the multi-level directory is determined based on the directory information in the request based on the S3 protocol.

[0065] During data storage or data reading, the client can send multiple requests, each of which triggers the engine layer to obtain a multi-level directory, and the index structure includes a multi-level index node corresponding to the multi-level directory, i.e., each directory corresponds to an index node. For example, based on the directory information in the request based on the S3 protocol, the first level directory is bucket1, the second level directory is dira, and the third level directory is file1, and the index structure can include a first level index node bucket1 (i.e., the directory name of the first level directory as the name of the first level index node, of course, other ways to define the name of the first level index node can also be used), a second level index node dira (i.e., the directory name of the second level directory as the name of the second level index node), and a third level index node file1. For another example, based on the directory information in the request based on the S3 protocol, the first level directory is bucket1, the second level directory is dirb, and the third level directory is file2, and the index structure can include a first level index node bucket1, a second level index node dirb, and a third level index node file2. For another example, based on the directory information in the request based on the S3 protocol, the first level directory is bucket1, the second level directory is dirc, the third level directory is dird, and the fourth level directory is file3, and the index structure can include a first level index node bucket1, a second level index node dirc, a third level index node dird, and a fourth level index node file3. For another example, based on the directory information in the request based on the S3 protocol, the first level directory is bucket1, and the second level directory is file4, and the index structure can include a first level index node bucket1 and a second level index node file4.

[0066] As can be seen from the above, based on the multi-level directory of the S3 protocol, the index structure shown in Fig. 4 can be obtained, which can include a first-level index node bucketl, second-level index nodes dira, dirb, dirc and file4, third-level index nodes filel, file2 and dird, and a fourth index node file3.

[0067] In one example, based on the index structure shown in Fig. 4, each index node can store metadata, and each index node only stores information of a next-level index node, and does not store information of an index node at another level across levels. For example, for each index node, if the index node is not a last-level index node of a multi-level index node, the metadata of the index node can include storage location information of a next-level index node of the index node, and if the index node is a last-level index node of a multi-level index node, the metadata of the index node can include storage location information of data to be stored.

[0068] For example, the storage location information of the first-level index node bucketl can be stored in a fixed storage location, and then the first-level index node bucketl is found based on the fixed storage location, and the metadata of the first-level index node bucketl is read. The metadata of the first-level index node bucketl includes storage location information of the second-level index nodes dira, dirb, dirc and file4. The second-level index node dira is found based on the storage location information of the second-level index node dira, and the metadata of the second-level index node dira is read. The second-level index node dirb is found based on the storage location information of the second-level index node dirb, and the metadata of the second-level index node dirb is read. The second-level index node dirc is found based on the storage location information of the second-level index node dirc, and the metadata of the second-level index node dirc is read. The second-level index node file4 is found based on the storage location information of the second-level index node file4, and the metadata of the second-level index node file4 is read.

[0069] The metadata of the second-level inode dira includes storage location information of the third-level inode filel, the third-level inode filel is found based on the storage location information of the third-level inode filel, and the metadata of the third-level inode filel is read. The metadata of the second-level inode dirb includes storage location information of the third-level inode file2, the third-level inode file2 is found based on the storage location information of the third-level inode file2, and the metadata of the third-level inode file2 is read. The metadata of the second-level inode dirc includes storage location information of the third-level inode dird, the third-level inode dird is found based on the storage location information of the third-level inode dird, and the metadata of the third-level inode dird is read. The metadata of the third-level inode dird includes storage location information of the fourth-level inode file3, the fourth-level inode file3 is found based on the storage location information, and the metadata of the fourth-level inode file3 is read.

[0070] The second-level inode file4 is the last-level inode, and the metadata of the second-level inode file4 includes storage location information of to-be-stored data (such as a file file4), indicating a storage location of the file file4. The third-level inode filel is the last-level inode, and the metadata of the third-level inode filel includes storage location information of to-be-stored data (such as a file filel), indicating a storage location of the file filel. The third-level inode file2 is the last-level inode, and the metadata of the third-level inode file2 includes storage location information of to-be-stored data (such as a file file2), indicating a storage location of the file file2. The fourth-level inode file3 is the last-level inode, and the metadata of the fourth-level inode file3 includes storage location information of to-be-stored data (such as a file file3), indicating a storage location of the file file3.

[0071] In summary, the engine layer can determine an index structure, which can include multiple levels of index nodes corresponding to the multiple levels of directories, and can determine metadata for each index node. For the metadata of each index node, the metadata of the index node can include storage location information of the next level index node of the index node. In addition, the metadata of the index node can also include inode (index node) information (such as storing attributes of a file, etc., such as permissions, type, size, time) and dentry (directory entry) information (such as the name of the index node, the name of the file, and inode information) of file semantics. In addition, the metadata of the index node can also include the owner of the S3 object, the etag (entity value of the requested variable, or the token associated with the Web resource), etc. For the coincident fields of different protocols, the metadata content of the index node can be reused. In addition, the content of the metadata can also be customized.

[0072] The persistence layer is used to store and read data and metadata based on the index structure, i.e., store and read data and metadata in a specified storage medium (such as a disk, etc.). For example, for each index node, store the metadata of the index node based on the storage location information of the index node; store the to-be-stored data based on the storage location information of the to-be-stored data. For example, for each index node, read the metadata of the index node based on the storage location information of the index node; read the to-be-stored data based on the storage location information of the to-be-stored data.

[0073] For example, store the metadata of the first level index node bucket1 at a fixed storage location of the disk, and read the metadata of the first level index node bucket1 at the fixed storage location of the disk.

[0074] The metadata of the first level index node bucket1 includes storage location information of the second level index nodes dira, dirb, dirc, and file4, the metadata of the second level index node dira is stored at the storage location information of the second level index node dira of the disk, the metadata of the second level index node dirb is stored at the storage location information of the second level index node dirb of the disk, and so on. Alternatively, the metadata of the second level index node dira is read from the storage location information of the second level index node dira of the disk, and the metadata of the second level index node dirb is read from the storage location information of the second level index node dirb of the disk.

[0075] The metadata of the second-level index node dira can include the storage location information of the third-level index node filel, and thus the metadata of the third-level index node filel can be stored at the storage location information of the third-level index node filel on the disk, and so on. Alternatively, the metadata of the third-level index node filel can be read from the storage location information of the third-level index node filel on the disk, and so on.

[0076] The metadata of the third-level index node filel can include the storage location information of the to-be-stored data (such as the file filel), and thus the file filel can be stored at the storage location information of the file filel on the disk. Alternatively, the metadata of the third-level index node filel can include the storage location information of the stored data (such as the file filel), and thus the file filel can be read from the storage location information of the file filel on the disk.

[0077] In an example, the to-be-stored data or the stored data can be unstructured data or other types of data, and the unstructured data can include, but is not limited to, social media content, images, audio, video, text, HTML, various reports, and the like, without limitation on the type of the unstructured data.

[0078] In an example, if the uniform resource locator in the request based on the object storage protocol (such as the S3 protocol) corresponds to the same multi-level directory as the file path in the request based on the distributed file storage protocol (such as the NFS protocol, the CIFS protocol, or the POSIX protocol), the uniform resource locator and the file path correspond to the same target index structure, that is, the metadata of the unstructured data and the unstructured data itself are written to the disk through the persistent layer, and the same file path or S3 URL points to the same data, that is, if the target index structure created based on the request based on the object storage protocol is the same as the target index structure created based on the request based on the distributed file storage protocol, the two target index structures point to the same unstructured data.

[0079] In summary, the objects or files created through the object storage protocol or the distributed file storage protocol and having the same index path can be represented by the same metadata, without being divided into two metadata. All requests from the protocol layer to the engine layer carry the current protocol type, facilitating the engine layer to perform differentiated and compatible processing according to the logic of the corresponding protocol semantics for the same metadata, and ensuring the completeness of the semantics.

[0080] The technical solutions of the present application are described below in combination with several specific application scenarios.

[0081] Application scenario 1: data storage process, which can include:

[0082] In step S11, a data storage request is obtained, the data storage request comprising directory information based on which a multi-level directory is determined, for example, the directory information can be split into the multi-level directory.

[0083] In an example, the protocol layer can receive a data storage request based on an object storage protocol (e.g., S3 protocol), and the data storage request comprises a uniform resource locator. Based on this, the uniform resource locator can be split into the multi-level directory based on a second separator in the uniform resource locator, i.e., each character after a second separator in the uniform resource locator corresponds to a directory, and the second separator can be “ / ”.

[0084] In an example, the protocol layer can receive a data storage request based on a distributed file storage protocol (e.g., NFS protocol, CIFS protocol, or POSIX protocol), and the data storage request can comprise a file path. Based on this, the file path can be split into the multi-level directory based on a first separator in the file path, i.e., each character after a first separator in the file path corresponds to a directory.

[0085] In step S12, a target index structure is determined based on the multi-level directory in the directory information, the target index structure comprising a plurality of index nodes corresponding to the multi-level directory. For example, the protocol layer can send the multi-level directory to the engine layer, and the engine layer determines the target index structure based on the multi-level directory after receiving the multi-level directory.

[0086] In an example, for each index node in the target index structure, if the index node is not a last-level index node of the multi-level index node, the metadata of the index node comprises storage location information of a next-level index node of the index node. Alternatively, if the index node is a last-level index node of the multi-level index node, the metadata of the index node comprises storage location information of the to-be-stored data.

[0087] In step S13, for each index node, the metadata of the index node is stored based on the storage location information of the index node; and the to-be-stored data is stored based on the storage location information of the to-be-stored data.

[0088] For example, the persistent layer is configured to store the to-be-stored data and the metadata (i.e., the metadata of the to-be-stored data) based on the target index structure, and the to-be-stored data can be unstructured data, such as storing the to-be-stored data and the metadata in a specified storage medium (e.g., a disk). For example, for each index node, the metadata of the index node is stored at the storage location information of the index node in the disk. In addition, for the to-be-stored data, the to-be-stored data is stored at the storage location information of the to-be-stored data in the disk.

[0089] Application scenario 2: data reading process, which can include:

[0090] Step S21, obtaining a data reading request including directory information, based on which a multi-level directory is determined, for example, the directory information can be divided into a multi-level directory.

[0091] In one example, the protocol layer can receive a data reading request based on an object storage protocol (such as S3 protocol), and the data reading request includes a uniform resource locator. Based on this, the uniform resource locator can be divided into a multi-level directory based on the second separator in the uniform resource locator, that is, each character after the second separator in the uniform resource locator corresponds to a directory, and the second separator can be “ / ”.

[0092] In one example, the protocol layer can receive a data reading request based on a distributed file storage protocol (such as NFS protocol, CIFS protocol or POSIX protocol), and the data reading request can include a file path. Based on this, the file path can be divided into a multi-level directory based on the first separator in the file path, that is, each character after the first separator in the file path corresponds to a directory.

[0093] Step S22, determining a target index structure based on the multi-level directory in the directory information, the target index structure including a multi-level index node corresponding to the multi-level directory. For example, the protocol layer can send the multi-level directory to the engine layer, and the engine layer determines the target index structure based on the multi-level directory after receiving the multi-level directory.

[0094] In one example, for each index node in the target index structure, if the index node is not the last level index node of the multi-level index node, the metadata of the index node includes the storage location information of the next level index node of the index node. Or, if the index node is the last level index node of the multi-level index node, the metadata of the index node includes the storage location information of the stored data.

[0095] Step S23, for each index node, reading the metadata of the index node based on the storage location information of the index node. If the index node is not the last level index node of the multi-level index node, determining the storage location information of the next level index node of the index node based on the metadata of the index node. If the index node is the last level index node of the multi-level index node, determining the storage location information of the stored data based on the metadata of the index node, and reading the stored data based on the storage location information.

[0096] For example, the persistent layer is used to read the stored data and metadata based on the target index structure, such as reading the stored data and metadata from a specified storage medium (such as a disk, etc.).

[0097] For example, for each index node, the storage location information of the index node is determined (i.e., the storage location information of the index node is obtained from the metadata of the upper-level index node of the index node), so that the metadata of the index node is read from the storage location information of the index node on the disk, and the metadata of the index node includes the storage location information of the lower-level index node of the index node, so that the storage location information of the lower-level index node of the index node can be determined based on the metadata.

[0098] By analogy, until the last index node of the multi-level index node, the metadata of the index node includes the storage location information of the stored data, so that the stored data can be read from the storage location information of the stored data on the disk, and the stored data is successfully read.

[0099] Application scenario 3: processing procedure of interface operation request based on distributed file storage protocol.

[0100] In an example, if an interface creation request based on the distributed file storage protocol is obtained, a new index node is added in the target index structure based on the interface creation request. For example, for an interface creation request based on the POSIX protocol, the interface creation request can be converted into an operation of adding an index node, so that a new index node can be added in the target index structure based on the interface creation request.

[0101] For example, the interface creation request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, and the target index structure includes multi-level index nodes corresponding to the multi-level directory. On this basis, the interface creation request includes information of an interface to be added, such as which directory the interface to be added is located in, so that a new index node can be added under the index node corresponding to the directory.

[0102] In an example, if an interface deletion request based on the distributed file storage protocol is obtained, an existing index node is deleted from the target index structure based on the interface deletion request. For example, for an interface deletion request based on the POSIX protocol, the interface deletion request can be converted into an operation of deleting an index node, so that an existing index node can be deleted from the target index structure based on the interface creation request.

[0103] For example, the interface deletion request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, and the target index structure includes multi-level index nodes corresponding to the multi-level directory. On this basis, the interface deletion request includes information of an interface to be deleted, such as which directory the interface to be deleted corresponds to, so that an index node corresponding to the directory can be deleted from the target index structure.

[0104] In one example, if a hard link interface request based on a distributed file storage protocol is obtained, a new inode is added in a target index structure, and the new inode points to an existing inode. For example, for a hard link interface request based on the POSIX protocol, the hard link interface can be converted to add a new inode, and the new inode points to an existing inode.

[0105] For example, the hard link interface request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, and the target index structure includes a plurality of inodes corresponding to the plurality of levels of the directory. On this basis, the hard link interface request includes information of an interface to be added, such as which directory the interface to be added is located in, and thus a new inode can be added under the inode corresponding to the directory, and the new inode points to an existing inode, to implement the hard link interface.

[0106] In one example, if a soft link interface request based on a distributed file storage protocol is obtained, a new inode is added in a target index structure, and the metadata of the new inode includes a string pointing to an arbitrary path. For example, for a soft link interface request based on the POSIX protocol, the soft link interface can be converted to add a new inode, and the metadata of the new inode includes a string pointing to an arbitrary path.

[0107] For example, the soft link interface request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, and the target index structure includes a plurality of inodes corresponding to the plurality of levels of the directory. On this basis, the soft link interface request includes information of an interface to be added, such as which directory the interface to be added is located in, and thus a new inode can be added under the inode corresponding to the directory, and the metadata of the new inode includes a string pointing to an arbitrary path, to implement the soft link interface.

[0108] In one example, if an interface write request based on a distributed file storage protocol is obtained, data is written to an offset address of an inode of a target index structure based on the interface write request. For example, for an interface write request based on the POSIX protocol, the interface write request can be converted to write a data block to an offset specified by the inode, and thus data can be written to the offset address of the inode based on the interface write request.

[0109] For example, the interface write request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, the target index structure including multi-level index nodes corresponding to the multi-level directory. On this basis, the interface write request includes information of data to be written, such as which directory the data to be written is located in, and thus the data can be written at an offset address specified by an index node corresponding to the directory.

[0110] In one example, if an interface read request based on a distributed file storage protocol is obtained, data is read from an offset address specified by an index node of a target index structure based on the interface read request. For example, for an interface read request of a POSIX protocol, the interface read request can be converted into reading a data block from an offset specified by the index node, and thus data can be read from the offset address specified by the index node based on the interface read request.

[0111] For example, the interface read request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, the target index structure including multi-level index nodes corresponding to the multi-level directory. On this basis, the interface read request includes information of data to be read, such as which directory the data to be read is located in, and thus the data can be read from an offset address specified by an index node corresponding to the directory.

[0112] Application scenario 4: processing procedure of an interface operation request for an object storage protocol (S3 protocol).

[0113] If an interface read request based on an object storage protocol (such as a get Object interface request) is obtained, the interface read request can include directory information, based on which a multi-level directory is determined, based on which a target index structure is determined, the target index structure including multi-level index nodes corresponding to the multi-level directory.

[0114] For each index node, metadata of the index node is obtained. If the index node is not the last level index node of the multi-level index node, storage location information of the next level index node of the index node can be determined based on the metadata of the index node, and metadata of the next level index node is obtained based on the storage location information, and so on, until the last level index node of the multi-level index node.

[0115] If the index node is the last level index node of the multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a hard link file (if the directory information of the interface read request points to a hard link created by the file scenario, the metadata of the index node includes information that the current data to be read is a hard link file), the storage location information of the stored data is determined based on the metadata of the index node, and the stored data is read based on the storage location information. For example, the data block can be read according to the OID (Object Identifier) in the metadata of the index node, and the reading process is not limited.

[0116] If the index node is the last level index node of the multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a soft link file (if the directory information of the interface read request points to a soft link created by the file scenario, the metadata of the index node includes information that the current data to be read is a soft link file), the path information of the soft link file is determined based on the metadata of the index node, and the new index node is queried according to the path information of the soft link file, the storage location information of the stored data is determined based on the metadata of the new index node, and the stored data is read based on the storage location information.

[0117] For example, for a hard link created by the file scenario, the metadata of the last level index node includes an OID, and the data block can be read based on the OID in the metadata of the index node. For a soft link created by the file scenario, the metadata of the last level index node includes path information of the soft link file, the new index node can be queried based on the path information of the soft link file in the metadata of the index node, and the metadata of the new index node includes an OID, and the data block can be read based on the OID in the metadata of the new index node.

[0118] In one example, the hard link is implemented as follows: an existing index node is specified, a new index node is created to point to the old index node, and the new index node does not carry any metadata, but only carries an OID pointing to the old index node. The soft link is implemented as follows: a new index node is created, and the path string of the input parameter is written into the symlink field of the current new index node.

[0119] Application scenario 5: processing process of interface operation request of object storage protocol (S3 protocol).

[0120] If a put interface request based on the object storage protocol is obtained, the put interface request is used to write a new object, and the put interface request can include directory information, then the multi-level directory is determined based on the directory information, the target index structure is determined based on the multi-level directory, and the target index structure includes the multi-level index node corresponding to the multi-level directory.

[0121] For each index node, the metadata of the index node is obtained. If the index node is not the last level index node of the multi-level index node, the storage location information of the next level index node of the index node can be determined based on the metadata of the index node, and the metadata of the next level index node is obtained based on the storage location information, and so on, until the last level index node of the multi-level index node.

[0122] If the index node is the last level index node of the multi-level index node, the data is written first, and after the data is written, the index node is created. For example, the put interface first applies a temporary OID, uses the temporary OID to write the data block, and finally updates the metadata of the index node. When updating the metadata of the index node, a new index node is created according to the temporary OID and the directory information.

[0123] Application scenario 6: processing procedure of interface operation request based on distributed file storage protocol.

[0124] If a rename interface request based on the distributed file storage protocol is obtained, the rename interface request can include directory information, and then the multi-level directory is determined based on the directory information, the target index structure is determined based on the multi-level directory, and the target index structure includes the multi-level index node corresponding to the multi-level directory.

[0125] The rename interface request can include information of the source index node (source directory), and therefore the source index node is deleted from the multi-level index node of the target index structure. The rename interface request can include information of the destination index node (destination directory), and therefore the destination index node is created in the target index structure.

[0126] The rename interface request can be submitted in two segments, which can ensure the atomicity of deleting the old index node and adding the new index node, that is, the first submission of the rename interface request includes the information of the source index node, and the second submission of the rename interface request includes the information of the destination index node.

[0127] In order to prevent the processing procedure of the rename interface request from conflicting with other interface operations, the metadata of the index node is marked when various modification interfaces are executed. The mark indicates a modification state, and at this time, other operations are not allowed to modify the index node, so as to prevent data disorder caused by concurrent modification.

[0128] Application scenario 7: processing procedure of special semantics of object storage protocol (S3 protocol).

[0129] ListObjects is a special semantic of object storage protocol, which is used to list all objects in a bucket, or filter or group all objects in a bucket according to a prefix and a delimiter. The semantic of ListObjects is similar to readdir of distributed file storage protocol, but ListObjects needs to output in lexicographical order while readdir of distributed file storage protocol does not limit the output order.

[0130] Based on this, if a ListObjects request based on object storage protocol is obtained, the ListObjects request can include directory information, then a multi-level directory is determined based on the directory information, a target index structure is determined based on the multi-level directory, the target index structure includes a multi-level index node corresponding to the multi-level directory, and the multi-level index node is a multi-level index node corresponding to a specified bucket in the ListObjects request.

[0131] For each index node, the metadata of the index node is obtained. If the index node is not the last level index node of the multi-level index node, the storage location information of the next level index node of the index node can be determined based on the metadata of the index node, the metadata of the next level index node is obtained based on the storage location information, and so on, until the last level index node of the multi-level index node.

[0132] The metadata of the multi-level index node corresponding to the bucket is returned to the client in a specified order. For example, the specified order can include but is not limited to lexicographical order, and the lexicographical order can be arranged in alphabetical order.

[0133] For example, if ListObjects is used to list all objects in a bucket, the metadata of all index nodes corresponding to the bucket can be returned to the client in the specified order. Or, if ListObjects is used to filter all objects in a bucket, the metadata of all index nodes corresponding to the bucket can be filtered, and the metadata of the filtered index nodes can be returned to the client in the specified order, and the filtering process is not limited. Or, if ListObjects is used to group all objects in a bucket, the metadata of all index nodes corresponding to the bucket can be grouped, and the metadata of the grouped index nodes can be returned to the client in the specified order, and the grouping process is not limited.

[0134] In one example, when storing the metadata of the multi-level index node, the metadata of the multi-level index node can be stored in a specified order. Based on this, when obtaining the metadata of the multi-level index node corresponding to the storage bucket, the metadata of the multi-level index node corresponding to the storage bucket can also be obtained in the specified order.

[0135] In one example, since the metadata has been split into multiple index nodes for storage, in the processing process for ListObjects, the split operation can be performed on the multi-layer index nodes in the storage bucket, and each layer of index node performs a readdir-like operation processing, but requires returning the result in lexicographic order. Finally, the sorting processing of multiple returned results is performed at the protocol layer to obtain the sorting result in the entire storage bucket.

[0136] In one example, in the metadata distribution management of the engine layer, the metadata of each storage node can be stored in the distributed database in lexicographic order, and then the metadata of each storage node is cached in the memory in a hash sorting manner. The reason for using the above manner is that in the file scenario, the hash sorting of the memory can be used to perform readdir processing and return faster, and in the listobject scenario, the lexicographic order of the distributed database can be used for processing and returning, thereby meeting the query interface capability of two different sorting manners.

[0137] When using the lexicographic order of the distributed database for processing and returning, the interface of the distributed database can be directly called for querying, such as returning the first 1000 results, returning the 1000-2000th result, etc.

[0138] Application scenario 8: processing process for special semantics of object storage protocol (S3 protocol).

[0139] Segmented (Multipart) upload is a special semantics of the object storage protocol. Segmented upload allows larger objects to be divided into multiple parts for uploading, and then these parts are combined into a complete object. The upper interface usually refers to a high-level interface for managing and controlling segmented upload, so that developers can more conveniently operate the segmented upload process, and lossless fusion needs to be compatible with the application scenario of segmented upload.

[0140] For the multipart upload process, after the client successfully initiates an InitMultipart request, the client uploads a batch of Parts (data segments) through multiple PUT requests. In the object storage protocol, the size of each Part can be different (the allowed Part size range is [5 MB-5 GB], and the last Part can be smaller than 5 MB), a segmented object has at most 10,000 Parts, and the Parts are allowed to be uploaded out of order (for example, the upload order can be 3, 2, 5, 4, 1). After all the Parts are uploaded, the CompleteMultipart request can be used to combine all the Parts into one large object.

[0141] As can be seen from the above, the upload steps of the segmented object are as follows: 1. InitMulitpart: generate an uploadid (upload identifier), apply for an OID of a data block, and store the event information of the Multipart to a distributed database.

[0142] 2. PutPart (upload data segment): write data to the data block corresponding to the OID, and after the data is written, generate the chunk information of the part, which includes OID + [part number] + [cover identifier] + [chunk number]. The OID represents the unique identifier of the segmented object, and all data segments (Parts) correspond to the same OID.

[0143] The part number represents the identifier of the data segment, for example, when the part number is 3, it represents the third data segment.

[0144] The cover identifier of the data segment represents the number of repetitions of the data segment. The cover identifier can be 0 by default. For the scenario of covering the same part, the [cover identifier] can be incremented by 1 each time the data is overwritten. A single part can support M times (such as 8 times) of concurrent overwriting. For example, the same data segment can be sent repeatedly 8 times. For the first sent data segment, the cover identifier of the data segment is 1, for the second sent data segment, the cover identifier of the data segment is 2, and so on.

[0145] The chunk number represents the identifier of the data chunk. For example, the same data segment can be divided into multiple data chunks, and the sizes of different data chunks can be the same or different. For each data chunk of the same data segment, the chunk number represents the identifier of the data chunk.

[0146] 3. `CompleteMultipart`: Generates complete metadata corresponding to the multipart object. It constructs a mapping table of the [part number], [overwrite identifier], and [block number] of the part data fragments, storing this mapping table as an array in the inode's metadata. This mapping table serves as extended data for the metadata. Furthermore, the `state` identifier indicates that the current inode is a Multipart type inode. During the `CompleteMultipart` process, the event information for the Multipart corresponding to the OID can also be deleted, indicating the completion of the segmented upload process.

[0147] Based on this, if a segmented upload request based on the object storage protocol is obtained, the segmented upload request may include directory information. Then, multi-level directories are determined based on the directory information, and the target index structure is determined based on the multi-level directories. The target index structure may include multi-level index nodes corresponding to the multi-level directories.

[0148] For each index node, retrieve its metadata. If the index node is not the last level index node in a multi-level index node system, the storage location information of the next level index node can be determined based on its metadata. The metadata of the next level index node can then be retrieved based on this storage location information, and so on, until the last level index node in the multi-level index node system is reached.

[0149] If the index node is the last level index node in a multi-level index node system, the storage location information of the data to be stored can be determined based on the metadata of the index node. Based on the storage location information of the data to be stored, the data blocks in the segmented upload request are stored. The data to be stored may include multiple data fragments, and each data fragment may include at least one data block.

[0150] After storing the data chunks in the segmented upload request, extended data is added to the metadata of the inode. This extended data includes the fragment identifier of the data fragment to which the data chunk belongs, the overlay identifier of the data fragment, the chunk identifier of the data chunk, and the size of the data chunk. For example, if all data chunks have the same size, which is a fixed value, the extended data may not include the size of the data chunks.

[0151] For example, after storing data block 1 of data shard 2 to disk, add extended data to the metadata of the last-level index node. This extended data includes the shard identifier 2 of data shard 2, the overlay identifier 1 of data shard 2, the block identifier 21 of data block 1, and the size of data block 1.

[0152] After storing the data chunk 2 of the data shard 2 to the disk, extended data is added in the metadata of the last level inode, and the extended data includes the shard identification 2 of the data shard 2, the coverage identification 1 of the data shard 2, the chunk identification 22 of the data chunk 2, and the size of the data chunk 2.

[0153] After storing the data chunk 1 of the data shard 1 to the disk, extended data is added in the metadata of the last level inode, and the extended data includes the shard identification 1 of the data shard 1, the coverage identification 1 of the data shard 1, the chunk identification 11 of the data chunk 1, and the size of the data chunk 1.

[0154] In this way, after storing each data chunk to the disk, extended data corresponding to the data chunk needs to be added in the metadata of the last level inode, and the order of the data shards is not limited, that is, the data shards are not stored in the order of 1, 2, 3, 4, and 5. In addition, the order of the data chunks of the same data shard is also not limited, that is, the data chunks are not stored in the order of 1, 2, 3, 4, and 5. Based on this, in the metadata of the last level inode, the order of the data shards is not limited, and the order of the data chunks is also not limited.

[0155] After storing all data shards to the disk, the segmented upload process is completed, and the metadata of the last level inode includes extended data corresponding to all data chunks of all data shards, and all extended data constitutes a mapping table, and all extended data is stored in the metadata of the inode in the form of an array.

[0156] After storing all data shards to the disk, a state identification can be added in the metadata of the last level inode, and the state identification indicates that the current inode is a Multipart type inode.

[0157] In an example, for the segmented read process, the client can initiate a read request based on the distributed file storage protocol (object storage protocol get request, the processing manners of the two are similar, and the read request is taken as an example for description in the following). Based on this, if the read request based on the distributed file storage protocol is obtained, the read request can include directory information, the multi-level directory is determined based on the directory information, and the target index structure is determined based on the multi-level directory, and the target index structure can include the multi-level index nodes corresponding to the multi-level directory.

[0158] For each index node, metadata of the index node is obtained. If the index node is not the last level index node of the multi-level index node, the storage location information of the next level index node of the index node can be determined based on the metadata of the index node, and the metadata of the next level index node is obtained based on the storage location information, and so on, until the last level index node of the multi-level index node.

[0159] If the index node is the last level index node of the multi-level index node, and the metadata of the index node includes the state identifier, it can be determined that the current index node is an Multipart type index node, that is, the read request is used to access the multipart object, and the segmented reading process needs to be performed.

[0160] Then, the target shard identifier of the data shard to be read is obtained based on the read request. For example, the read request can include the shard identifier of the data shard to be read, and the shard identifier is used as the target shard identifier, or other ways can be used to determine the target shard identifier, which is not limited.

[0161] Then, the maximum coverage identifier corresponding to the target shard identifier is determined based on the metadata of the index node. For example, all extension data corresponding to the target shard identifier is traversed in sequence, and the maximum coverage identifier can be determined from the extension data, and the maximum coverage identifier is used to represent the coverage identifier of the last sent data shard. For example, it is assumed that the extension data corresponding to the target shard identifier includes coverage identifiers 1, 2 and 3, and the maximum coverage identifier is coverage identifier 3, that is, the data shard is repeatedly sent 3 times.

[0162] Then, the target extension data corresponding to the target shard identifier and the maximum coverage identifier is determined from the metadata of the index node. For example, it is assumed that the data shard corresponds to 2 data chunks, the target shard identifier is shard identifier 2, the maximum coverage identifier is coverage identifier 3, and the target extension data includes extension data of shard identifier 2+coverage identifier 3+chunk identifier 21, and extension data of shard identifier 2+coverage identifier 3+chunk identifier 22.

[0163] Then, the offset and length of the data shard to be read are determined based on the sorting position of the target extension data and the size of the data chunk in the target extension data. The offset represents the offset of the data shard to be read in the disk, and the length represents the data length of the data shard to be read.

[0164] For example, for the extended data of the shard identifier 2 + the overlay identifier 3 + the chunk identifier 21, assuming that the order position of the extended data is 6 (i.e., the 6th extended data), and the total length of the previous 5 extended data is A (i.e., the sum of the sizes of the data chunks in the 5 extended data is A), the offset address of the to-be-read data shard is A, and the length of the to-be-read data shard is the size of the data chunk in the extended data.

[0165] Then, based on the metadata of the index node, the storage location information corresponding to the to-be-read data shard is determined (the metadata of the last level index node includes the storage location information), and based on the storage location information, the offset address and the length, the to-be-read data shard can be obtained. For example, starting from the first address of the storage location information, the position after the offset address is taken as the start address of the to-be-read data shard, and the data of the length is read from the start address, and this part of data is the to-be-read data shard.

[0166] At this point, the reading process of the to-be-read data shard is completed, that is, the actually hit data shard can be found according to the offset address (offset) and the length (len) for reading. If the to-be-read data shard includes multiple data shards, the results of multiple data shards need to be read and spliced, and then returned to the client.

[0167] In one example, for the overlay write scenario, the client can initiate an overlay write request based on the distributed file storage protocol. The processing mode of the overlay write request is similar to that of the read request, and the actual storage location of the to-be-written data shard needs to be determined based on the storage location information, the offset address and the length of the to-be-written data shard. Unlike the read request, instead of reading the to-be-read data shard at the actual storage location, the to-be-written data shard in the overlay write request is updated to the actual storage location, thereby covering the existing data shard. In addition, if overlay write needs to be performed across multiple data shards, it can be split into multiple data shards for writing, and then the metadata of the index node is updated.

[0168] In one example, for the truncate scenario of the distributed file storage protocol, the size of the file can be truncated to a specified length, and when a truncate operation is performed on a file, the data at the tail of the file is deleted, and the size of the file is reduced to the specified length. Based on this, for the truncation process, the client can initiate a truncate operation request based on the distributed file storage protocol. If the truncate operation request based on the distributed file storage protocol is obtained, the truncate operation request can include directory information, then the multi-level directory is determined based on the directory information, and the target index structure is determined based on the multi-level directory, and the target index structure can include the multi-level index nodes corresponding to the multi-level directory.

[0169] For each index node, metadata of the index node is obtained. If the index node is not the last level index node of the multi-level index node, the storage location information of the next level index node of the index node can be determined based on the metadata of the index node, and the metadata of the next level index node is obtained based on the storage location information, and so on, until the last level index node of the multi-level index node.

[0170] If the index node is the last level index node of the multi-level index node, and the metadata of the index node includes the state identifier, it can be determined that the current index node is an index node of the Multipart type, that is, the truncation operation request is used for the truncation operation on the multipart object.

[0171] Then, the extension data of a plurality of data shards (such as all data shards) can be determined from the metadata of the index node, and the plurality of data shards are sorted based on the extension data. The sorting result can be, for example, data shard 1 (the data shard with a shard identifier of 1), data shard 2, and so on.

[0172] Based on the sorting result and the truncation size in the truncation operation request, the data shards to be truncated are determined. For example, K data shards are selected from data shard 1, and the total size of the K data shards (that is, the total size of all data chunks of the K data shards) can be equal to the truncation size in the truncation operation request. The last data shard (the Kth data shard) can be a complete data shard or an incomplete data shard. It is necessary to ensure that the total size of the K data shards is equal to the truncation size, so as to truncate all data shards using the truncation size, and then the K data shards are taken as the data shards to be truncated.

[0173] For each data shard to be truncated, the offset address and the length of the data shard to be truncated are determined based on the sorting position of the extension data of the data shard to be truncated and the size of each data chunk in the data shard to be truncated. The offset address represents the offset address of the data shard to be truncated in the disk, and the length represents the data length of the data shard to be truncated. For example, if the data shard to be truncated includes a plurality of data chunks, and the plurality of data chunks are continuous, the offset address represents the offset address of the first data chunk in the disk, and the length represents the data length of all data chunks. For example, if the data shard to be truncated includes a plurality of data chunks, and the plurality of data chunks are not continuous, each data chunk corresponds to an offset address and a length separately. The offset address represents the offset address of the data chunk in the disk, and the length represents the data length of the data chunk.

[0174] Then, based on the metadata of the index node, the storage location information corresponding to the to-be-truncated data segment is determined (the metadata of the last index node includes the storage location information), and based on the storage location information, the offset address and the length, the to-be-truncated data segment is subjected to the emptying operation. For example, starting from the first address of the storage location information, the position after the offset address is taken as the start address of the to-be-truncated data segment, and the data of the length starting from the start address is the to-be-truncated data segment. Based on this, the to-be-truncated data segment can be subjected to the emptying operation.

[0175] In summary, when truncate is performed on a multipart object, the size to be truncated (truncation size) is used to find the segment information (to-be-truncated data segment) involved, and data truncation is performed segment by segment, i.e., the to-be-truncated data segment is subjected to the emptying operation, and the critical segment may truncate part of the data in the segment.

[0176] Application scenario 9: processing process for special semantics of an object storage protocol (S3 protocol).

[0177] The multi-version function is a special semantics of the object storage protocol, and the multi-version function allows storing multiple versions of objects on the same key (Key). Thus, when a write operation occurs, the object storage protocol does not directly overwrite the old version of the object, but retains it, thereby forming a version history record. Each version of the object has a unique version identifier when stored. The multi-version function can provide a data protection mechanism, i.e., even if an unexpected write operation occurs, the previous version can be restored. Based on this, the metadata of the index node needs to consider the snapshot capability and the multi-version capability. For example, the metadata of the index node includes a snapshot identifier, and when an object of the object storage protocol or a file of the distributed file storage protocol is uploaded, a snapshot identifier is backed up as the latest version and added to the metadata, and the snapshot identifier is incremental and greater than 0. Based on this multi-version feature, the snapshot identifier in the metadata of the index node is assigned to a specific value (such as a negative value, such as -1, -2, -3, etc.), and the state identifier is marked, indicating that the index node has the multi-version feature.

[0178] On this basis, in the case of opening multiple versions, for each index node, if the metadata of the index node includes the state identifier (that is, the index node has the multiple version feature), then: when generating the first metadata of the current version (that is, the latest version) for the index node, the latest version identifier (that is, generating a version identifier as the latest version identifier, and the latest version identifier has uniqueness) can be generated for the first metadata, and the latest version identifier is added in the first metadata. And, if the index node has corresponding second metadata of the historical version, a snapshot identifier can be generated for the second metadata, and the snapshot identifier can be a negative value (such as -1, -2, -3, etc.), and the snapshot identifier is added for the second metadata, a target version identifier is generated for the second metadata, and the target version identifier is added for the second metadata.

[0179] For example, if an upload request based on an object storage protocol is received, it is determined that the first metadata of the current version needs to be generated for the index node, and the index node has the second metadata (that is, the old metadata) corresponding to the historical version. Based on this, the snapshot identifier is generated for the second metadata, and the snapshot identifier is added to the second metadata, so that the second metadata is as a historical version. The first metadata is generated for the index node, that is, the first metadata is as a current version. In this way, there are two metadata with different versions, which meets the semantics of multiple versions.

[0180] In one example, when generating the target version identifier for the second metadata of the historical version, the target version identifier can include a first random character, a target ordered character, and a second random character, the target ordered character can be obtained by performing an encryption operation on the initial ordered character and the first random character, and in addition, the initial ordered character can be obtained by performing an operation on the initial ordered character of the metadata of the previous version.

[0181] For example, the target version identifier (which can also be referred to as the version number of the multiple version, denoted as versionId) is a string of 31 characters, and the multiple version object is divided into a historical version, a current version, and a delete marker.

[0182] For example, the target version identifier can include an 11-bit first random character, an 8-bit encrypted target ordered character, and a 12-bit second random character, which is only an example, and the target version identifier is not limited. The 11-bit first random character and the 12-bit second random character can be randomly set.

[0183] For example, an 8-bit ordered character can be given an initial value, such as z9JH27Gz, the initial ordered character (order_str) of the first version can be z9JH27Gz, and the initial ordered character of each subsequent new version is one less than the initial ordered character of the previous version in lexicographic order, that is, the initial ordered character (order_str) of the second version can be z9JH27Gy, and so on. Thus, the initial ordered character of each version can be obtained, that is, the initial ordered character of the current version can be obtained by performing an operation on the initial ordered character of the previous version of metadata, that is, one less than the initial ordered character of the previous version in lexicographic order.

[0184] Based on the initial ordered character of the current version and the first 11-bit random character, an encrypted 8-bit target ordered character can be calculated, for example, the target ordered character can be [order_str]⊕[the first 8 bits of the 11-bit random character], and order_str represents the initial ordered character of the current version. In summary, the target ordered character can be obtained by performing an encryption operation on the initial ordered character and the first random character.

[0185] For example, referring to FIG. 5, a schematic diagram of a target version identifier is shown. For the first version, the initial ordered character is z9JH27Gz, and the first 11-bit random character and the second 12-bit random character can be randomly generated. Then, the initial ordered character and the first 8 bits of the first random character are subjected to base16 XOR and then base64 conversion to obtain an encrypted 8-bit target ordered character. Then, the first 11-bit random character, the 8-bit target ordered character, and the second 12-bit random character can be combined to obtain the target version identifier.

[0186] For the second version, the initial ordered character is z9JH27Gy (determined based on the initial ordered character of the first version), and the first 11-bit random character and the second 12-bit random character can be randomly generated. Then, the initial ordered character and the first 8 bits of the first random character are subjected to base16 XOR and then base64 conversion to obtain an encrypted 8-bit target ordered character. Then, the first 11-bit random character, the 8-bit target ordered character, and the second 12-bit random character can be combined to obtain the target version identifier.

[0187] Based on the above processing mode, the version number of the multiple versions is assigned to the ordered attribute, the functions of multiple version ordering and historical object access are implemented using an index, and the storage space and logical implementation complexity are saved. When accessing in a distributed file storage protocol (such as NFS, CIFS, POSIX), only the metadata of the current version is seen, the readdir, stat and other interfaces of the file are filtered out, the inode of the snapshot identification is negative, the multiple version object with invisible file semantics is not affected, and the normal use of the file is not affected, so that compatibility is achieved.

[0188] In the file snapshot scenario, the snapshot identification allocated when the file is snapped is a positive number, which is incremented from 1, and therefore, the snapshot identification of the file accessed in the snapshot version and the identification of the object multiple version are completely different and can be easily distinguished, so that the two multiple versions and snapshot features are not interfered with each other and are compatible with each other.

[0189] In an example, in the permission control of different protocols, in order to ensure the completeness of the semantics of multiple protocols, the same protocol semantics are combined for processing. For example, in the distributed file storage protocol, the NFSV3 and POSIX semantics are controlled by ugo and POSIX ACL, and then, the places with semantic differences, such as NFSV4 ACL, CIFS using NTACL, S3 object ACL and bucket ACL, are not integrated, but the original semantic logic is retained, and the control fields of these ACLs are persisted to the extended attributes of the corresponding inode metadata. Then, when the operation corresponding to the protocol is performed, the operation carried by the protocol layer to the engine layer carries the protocol type of the current operation, and the engine layer can read the ACL information of the inode metadata according to the corresponding protocol type to perform fine-grained permission control.

[0190] As can be seen from the above technical solutions, in the embodiments of the present application, the intercommunication of the object storage protocol and the distributed file storage protocol can be implemented, the semantic loss and performance attenuation can be reduced when different protocols are accessed, and the space utilization rate can be maximized. For example, a storage system supporting the intercommunication of S3, NFS, CIFS and POSIX protocols can be used to solve the problems of semantic loss and performance attenuation caused by the intercommunication of different protocols. In the multiple version scenario and the segmented upload scenario, the lossless intercommunication of the distributed file storage protocol and the object storage protocol can also be completely compatible with the readdir interface of the file and the listobject interface of the object.

[0191] Flexibility and compatibility: Supports NFS, suitable for Unix / Linux systems and multi-user environments. Supports CIFS / SMB (Server Message Block), suitable for Windows environments. Supports S3, suitable for modern cloud computing and big data environments. This multi-protocol support can meet the needs of different enterprises, different operating systems and applications can seamlessly access the same data storage, improving the flexibility and compatibility of business. Data intercommunication: Data can be intercommunicated between different protocols, whether accessed using NFS, CIFS or S3 protocol, users can see the same data, this intercommunication simplifies data management and data sharing, reduces the problem of data silos. Consistency guarantee: Seamless data access and sharing means that users do not need to worry about data inconsistency when switching between different protocols, data changes are reflected in all protocol access paths in real time. Cost savings: No need to maintain multiple independent storage systems for different protocols, through unified storage solutions can significantly reduce hardware, software and operation and maintenance costs. Reduce data migration between multiple storage, shorten the entire process time and reduce equipment and management costs.

[0192] Simplify management: consistent data management interface and strategy make system administrators can more efficiently manage storage resources, unified management tools and strategies reduce complexity and management overhead. Improve productivity: business departments, teams can more efficiently collaborate and share data, regardless of which operating system or application is used. Enhance business continuity and data protection: unified storage solutions support advanced data protection functions such as snapshots, backups and disaster recovery, which can provide higher business continuity guarantee. Longer life cycle support: due to the reduction of hardware and software rapid aging caused by intercommunication restrictions, more flexible and reliable system prolongs the life cycle of storage solutions.

[0193] Based on the same application concept as the above method, the embodiment of the present application proposes a data storage device, as shown in Figure 6, which is a structural schematic diagram of the data storage device. The device can include:

[0194] The acquisition module 61 is used for acquiring a data storage request, and the data storage request includes directory information. If the data storage request is based on an object storage protocol, the directory information includes a uniform resource locator. If the data storage request is based on a distributed file storage protocol, the directory information includes a file path.

[0195] The determining module 62 is configured to determine a target index structure based on the directory information, the target index structure including a plurality of index nodes corresponding to a plurality of directories in the directory information; wherein, for each index node, if the index node is not a last level index node of the plurality of index nodes, metadata of the index node includes storage location information of a next level index node of the index node, and if the index node is the last level index node of the plurality of index nodes, the metadata of the index node includes storage location information of the to-be-stored data.

[0196] The storage module 63 is configured to, for each index node, store the metadata of the index node based on the storage location information of the index node; and store the to-be-stored data based on the storage location information of the to-be-stored data.

[0197] In one example, the obtaining module 61 is further configured to, if the directory information includes a file path, split the file path into a plurality of directories based on a first separator in the file path; or,

[0198] if the directory information includes a uniform resource locator, split the uniform resource locator into a plurality of directories based on a second separator in the uniform resource locator; wherein, characters after each second separator in the uniform resource locator correspond to a directory.

[0199] In one example, if a uniform resource locator in a storage request of an object storage protocol corresponds to a same plurality of directories as a file path in a storage request of a distributed file storage protocol, the uniform resource locator and the file path correspond to a same target index structure; the to-be-stored data is unstructured data; and the object storage protocol includes a simple storage service protocol, and the distributed file storage protocol includes at least one of a network file system protocol, a universal internet file system protocol, and a portable operating system interface protocol.

[0200] In one example, the obtaining module 61 is further configured to obtain a data reading request, the data reading request comprising directory information; wherein, if the data reading request is based on an object storage protocol, the directory information comprises a uniform resource locator; if the data reading request is based on a distributed file storage protocol, the directory information comprises a file path; the determining module 62 is further configured to determine a plurality of index nodes corresponding to a plurality of directories in the directory information; the apparatus further comprises a processing module configured to, for each index node, read metadata of the index node based on storage location information of the index node; if the index node is not a last index node of the plurality of index nodes, determine storage location information of a next index node of the index node based on the metadata of the index node; if the index node is the last index node of the plurality of index nodes, determine storage location information of stored data based on the metadata of the index node, and read the stored data based on the storage location information.

[0201] In one example, the obtaining module 61 is further configured to, if an interface creation request based on a distributed file storage protocol is obtained, add a new index node in the target index structure based on the interface creation request; if an interface deletion request based on a distributed file storage protocol is obtained, delete an existing index node from the target index structure based on the interface deletion request; if a hard link interface request based on a distributed file storage protocol is obtained, add a new index node in the target index structure, and point the new index node to an existing index node; if a soft link interface request based on a distributed file storage protocol is obtained, add a new index node in the target index structure, and metadata of the new index node comprises a string pointing to an arbitrary path; if an interface write request based on a distributed file storage protocol is obtained, write data to an offset address specified by an index node of the target index structure based on the interface write request; if an interface read request based on a distributed file storage protocol is obtained, read data from an offset address specified by an index node of the target index structure based on the interface read request.

[0202] In one example, the obtaining module 61 is further configured to, if the interface read request based on the object storage protocol is obtained, determine a multi-level index node based on the interface read request; for each index node, obtain metadata of the index node; the determining module 62 is further configured to, if the index node is the last level index node of the multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a hard link file, determine storage location information of the stored data based on the metadata of the index node, read the stored data based on the storage location information; or, if it is determined based on the metadata of the index node that the interface read request is used to read a soft link file, determine path information of the soft link file based on the metadata of the index node, and query a new index node according to the path information of the soft link file, determine storage location information of the stored data based on the metadata of the new index node, and read the stored data based on the storage location information.

[0203] In one example, the obtaining module 61 is further configured to, if the list object request based on the object storage protocol is obtained, the list object request is used to list all objects in the storage bucket, or the list object request is used to filter or group all objects in the storage bucket according to a prefix and a separator, determine the multi-level index node corresponding to the storage bucket based on the list object request; for each index node, obtain metadata of the index node; return the metadata of the multi-level index node corresponding to the storage bucket to the client in a specified order; wherein, when storing the metadata of the multi-level index node, store the metadata of the multi-level index node in the specified order; when obtaining the metadata of the multi-level index node corresponding to the storage bucket, obtain the metadata of the multi-level index node corresponding to the storage bucket in the specified order; wherein, the specified order includes a lexicographical order, and the lexicographical order is arranged in alphabetical order.

[0204] In one example, the obtaining module 61 is further configured to, if the segmented upload request based on the object storage protocol is obtained, determine a multi-level index node based on the segmented upload request; for each index node, obtain metadata of the index node; the determining module 62 is further configured to, if the index node is the last level index node of the multi-level index node, determine storage location information of the to-be-stored data based on the metadata of the index node; the storage module 63 is further configured to store the data blocks in the segmented upload request based on the storage location information of the to-be-stored data; wherein the to-be-stored data includes a plurality of data segments, and each data segment includes at least one data block; the storage module 63 is further configured to, after storing the data blocks in the segmented upload request, add extension data in the metadata of the index node, wherein the extension data includes a segment identifier of the data segment to which the data block belongs, an overlay identifier of the data segment, a block identifier of the data block, and a size of the data block; and wherein the overlay identifier of the data segment indicates a number of repetitions of the data segment.

[0205] In one example, the obtaining module 61 is further configured to, if the read request based on the distributed file storage protocol or the get request based on the object storage protocol is obtained, determine a multi-level index node based on the read request or the get request; for each index node, obtain metadata of the index node; the determining module 62 is further configured to, if the index node is the last level index node of the multi-level index node, obtain a target segment identifier of a to-be-read data segment based on the read request or the get request; determine a maximum overlay identifier corresponding to the target segment identifier based on the metadata of the index node, determine target extension data corresponding to the target segment identifier and the maximum overlay identifier from the metadata of the index node; determine an offset address and a length of the to-be-read data segment based on a sorting position of the target extension data and a size of a data block in the target extension data; determine storage location information corresponding to the to-be-read data segment based on the metadata of the index node, and obtain the to-be-read data segment based on the storage location information, the offset address, and the length.

[0206] In one example, the obtaining module 61 is further configured to, if the truncated operation request based on the distributed file storage protocol is obtained, determine a multi-level index node based on the truncated operation request; for each index node, obtain metadata of the index node; the determining module 62 is further configured to, if the index node is a last level index node of the multi-level index node, determine extended data of a plurality of data shards from the metadata of the index node, sort the plurality of data shards based on the extended data; determine a data shard to be truncated based on a result of the sorting and a truncation size in the truncated operation request; determine an offset address and a length of the data shard to be truncated based on a sorting position of the extended data of the data shard to be truncated and sizes of data blocks in the data shard to be truncated; determine storage location information corresponding to the data shard to be truncated based on the metadata of the index node, and perform a clearing operation on the data shard to be truncated based on the storage location information, the offset address and the length.

[0207] In one example, the processing module is configured to, for each index node, generate a latest version identifier for first metadata of a current version of the index node when the first metadata is generated, add the latest version identifier in the first metadata; if the index node has corresponding second metadata of a historical version, generate a snapshot identifier for the second metadata, the snapshot identifier being a negative value, add the snapshot identifier for the second metadata, generate a target version identifier for the second metadata, and add the target version identifier for the second metadata; wherein the target version identifier includes a first random character, a target ordered character and a second random character, the target ordered character being obtained by performing an encryption operation on an initial ordered character and the first random character, and the initial ordered character being obtained by performing an operation on an initial ordered character of metadata of a previous version.

[0208] Based on the same application concept as the above method, an electronic device is provided in the embodiments of the present application, as shown in FIG. 7, the electronic device includes a processor 71 and a machine readable storage medium 72, the machine readable storage medium 72 stores machine executable instructions capable of being executed by the processor 71; the processor 71 is configured to execute the machine executable instructions to implement the data storage method disclosed in the above examples of the present application.

[0209] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by a processor to implement the data storage method disclosed in the above examples of the present application.

[0210] The machine-readable storage medium described above can be any electronic, magnetic, optical, or other physical storage device that contains or stores information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be a Random Access Memory (RAM), a volatile memory, a non-volatile memory, a flash memory, a storage drive, such as a hard drive, a solid-state drive, any type of storage disk (e.g., a floppy disk, a DVD, etc.), or any suitable storage medium, or a combination thereof.

[0211] Based on the same application concept as the method described above, the embodiments of the present application further provide a computer program product, which can include a computer program. When the computer program is executed by a processor, the data storage method disclosed in the embodiments of the present application is implemented.

[0212] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, a disk storage, a CD-ROM, an optical storage, etc.) containing computer-usable program code.

[0213] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A data storage method, characterized by, The method comprises: acquiring a data storage request, the data storage request comprising directory information; wherein, if the data storage request is based on an object storage protocol, the directory information comprises a uniform resource locator; if the data storage request is based on a distributed file storage protocol, the directory information comprises a file path; determining a target index structure based on the directory information, the target index structure comprising a plurality of index nodes corresponding to a plurality of directories in the directory information; wherein, for each index node, if the index node is not the last index node of the plurality of index nodes, the metadata of the index node comprises storage location information of the next index node of the index node, and if the index node is the last index node of the plurality of index nodes, the metadata of the index node comprises storage location information of the to-be-stored data; for each index node, storing the metadata of the index node based on the storage location information of the index node, and storing the to-be-stored data based on the storage location information of the to-be-stored data.

2. The method of claim 1, wherein, If the directory information comprises a file path, the file path is divided into a plurality of directories based on a first separator in the file path. Or, if the directory information comprises a uniform resource locator, the uniform resource locator is divided into a plurality of directories based on a second separator in the uniform resource locator; wherein, the characters after each second separator in the uniform resource locator correspond to a directory.

3. The method of claim 1, wherein, If the uniform resource locator in the storage request of the object storage protocol and the file path in the storage request of the distributed file storage protocol correspond to the same plurality of directories, the uniform resource locator and the file path correspond to the same target index structure. Wherein, the to-be-stored data is unstructured data; the object storage protocol comprises a simple storage service protocol, and the distributed file storage protocol comprises at least one of a network file system protocol, a universal internet file system protocol, and a portable operating system interface protocol.

4. The method of claim 1, wherein, After storing the to-be-stored data based on the storage location information of the to-be-stored data, the method further comprises: acquiring a data read request, the data read request comprising directory information; wherein, if the data read request is based on an object storage protocol, the directory information comprises a uniform resource locator; if the data read request is based on a distributed file storage protocol, the directory information comprises a file path; determining a plurality of index nodes corresponding to a plurality of directories in the directory information; for each index node, reading the metadata of the index node based on the storage location information of the index node; if the index node is not the last index node of the plurality of index nodes, determining the storage location information of the next index node of the index node based on the metadata of the index node; if the index node is the last index node of the plurality of index nodes, determining the storage location information of the stored data based on the metadata of the index node, and reading the stored data based on the storage location information.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: If an interface creation request based on a distributed file storage protocol is obtained, a new index node is added in the target index structure based on the interface creation request; If an interface deletion request based on a distributed file storage protocol is obtained, an existing index node is deleted from the target index structure based on the interface deletion request; If a hard link interface request based on a distributed file storage protocol is obtained, a new index node is added in the target index structure, and the new index node points to an existing index node; If a soft link interface request based on a distributed file storage protocol is obtained, a new index node is added in the target index structure, and the metadata of the new index node includes a string pointing to an arbitrary path If an interface write request based on a distributed file storage protocol is obtained, data is written to an offset address specified by an index node of the target index structure based on the interface write request; If an interface read request based on a distributed file storage protocol is obtained, data is read from an offset address specified by an index node of the target index structure based on the interface read request. The method further comprises:

6. The method according to any one of claims 1 to 4, characterized in that, If an interface read request based on an object storage protocol is obtained, a multi-level index node is determined based on the interface read request; for each index node, the metadata of the index node is obtained; If the index node is the last level index node of the multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a hard link file, the storage location information of the stored data is determined based on the metadata of the index node, and the stored data is read based on the storage location information; Or, if it is determined based on the metadata of the index node that the interface read request is used to read a soft link file, the path information of the soft link file is determined based on the metadata of the index node, a new index node is queried according to the path information of the soft link file, the storage location information of the stored data is determined based on the metadata of the new index node, and the stored data is read based on the storage location information. The method further comprises:

7. The method according to any one of claims 1 to 4, characterized in that, If a list object request based on an object storage protocol is obtained, the list object request is used to list all objects in a storage bucket, or the list object request is used to filter or group all objects in a storage bucket according to a prefix and a separator, a multi-level index node corresponding to the storage bucket is determined based on the list object request; for each index node, the metadata of the index node is obtained; The metadata of the multi-level index node corresponding to the storage bucket is returned to the client in a specified order; Wherein, when storing the metadata of the multi-level index node, the metadata of the multi-level index node is stored in the specified order; when obtaining the metadata of the multi-level index node corresponding to the storage bucket, the metadata of the multi-level index node corresponding to the storage bucket is obtained in the specified order; Wherein, the specified order includes a lexicographical order, and the lexicographical order is arranged in alphabetical order. The method further comprises:

8. The method according to any one of claims 1 to 4, characterized in that, If a segmented upload request based on an object storage protocol is obtained, a multi-level index node is determined based on the segmented upload request; for each index node, the metadata of the index node is obtained; ​ If the index node is the last index node of the multi-level index node, storage location information of to-be-stored data is determined based on metadata of the index node; data chunks in the segmented upload request are stored based on the storage location information of the to-be-stored data; wherein the to-be-stored data comprises a plurality of data shards, and for each data shard, the data shard comprises at least one data chunk; After storing the data chunks in the segmented upload request, extension data is added in the metadata of the index node, and the extension data comprises shard identification of a data shard to which the data chunk belongs, coverage identification of the data shard, chunk identification of the data chunk, and size of the data chunk. The coverage identification of the data shard indicates a repetition number of the data shard.

9. The method of claim 8, wherein, The method further comprises: If a read request based on a distributed file storage protocol or an object storage protocol get request is acquired, a multi-level index node is determined based on the read request or the get request; for each index node, metadata of the index node is acquired; If the index node is the last index node of the multi-level index node, target shard identification of a to-be-read data shard is acquired based on the read request or the get request; a maximum coverage identification corresponding to the target shard identification is determined based on the metadata of the index node, target extension data corresponding to the target shard identification and the maximum coverage identification is determined from the metadata of the index node; based on a sorting position of the target extension data and size of data chunks in the target extension data, an offset address and a length of the to-be-read data shard are determined; Based on the metadata of the index node, storage location information corresponding to the to-be-read data shard is determined, and the to-be-read data shard is acquired based on the storage location information, the offset address and the length. The method further comprises:

10. The method of claim 8, wherein, If a truncation operation request based on a distributed file storage protocol is acquired, a multi-level index node is determined based on the truncation operation request; for each index node, metadata of the index node is acquired; If the index node is the last index node of the multi-level index node, extension data of a plurality of data shards is determined from the metadata of the index node, and the plurality of data shards are sorted based on the extension data; based on a sorting result and a truncation size in the truncation operation request, a to-be-truncated data shard is determined; Based on a sorting position of extension data of the to-be-truncated data shard and size of each data chunk in the to-be-truncated data shard, an offset address and a length of the to-be-truncated data shard are determined; Based on the metadata of the index node, storage location information corresponding to the to-be-truncated data shard is determined, and a clear operation is performed on the to-be-truncated data shard based on the storage location information, the offset address and the length. The method further comprises:

11. The method according to any one of claims 1 to 4, characterized in that, For each index node, when generating first metadata of a current version for the index node, a latest version identification is generated for the first metadata, and the latest version identification is added in the first metadata; ​ If the inode corresponds to a second metadata of a historical version, a snapshot identifier is generated for the second metadata, the snapshot identifier is a negative value, the snapshot identifier is added to the second metadata, a target version identifier is generated for the second metadata, and the target version identifier is added to the second metadata; The target version identifier includes a first random character, a target ordered character, and a second random character, the target ordered character is obtained by performing an encryption operation on an initial ordered character and the first random character, and the initial ordered character is obtained by performing an operation on an initial ordered character of metadata of a previous version.

12. A data storage device, characterized by The apparatus includes: The obtaining module is configured to obtain a data storage request, the data storage request including directory information; if the data storage request is based on an object storage protocol, the directory information includes a uniform resource locator; if the data storage request is based on a distributed file storage protocol, the directory information includes a file path; The determining module is configured to determine a target index structure based on the directory information, the target index structure including a plurality of index nodes corresponding to a plurality of directories in the directory information; for each index node, if the index node is not a last-level index node of the plurality of index nodes, metadata of the index node includes storage location information of a next-level index node of the index node, and if the index node is the last-level index node of the plurality of index nodes, the metadata of the index node includes storage location information of to-be-stored data. The storage module is configured to, for each index node, store metadata of the index node based on storage location information of the index node; and store the to-be-stored data based on the storage location information of the to-be-stored data.

13. The apparatus of claim 12, wherein The obtaining module is further configured to, if the directory information includes the file path, split the file path into a plurality of directories based on a first separator in the file path; or if the directory information includes the uniform resource locator, split the uniform resource locator into a plurality of directories based on a second separator in the uniform resource locator; wherein a character after each second separator in the uniform resource locator corresponds to a directory.

14. The apparatus of claim 12, wherein, If a uniform resource locator in a storage request of the object storage protocol and a file path in a storage request of the distributed file storage protocol correspond to a same plurality of directories, the uniform resource locator and the file path correspond to a same target index structure. The to-be-stored data is unstructured data; the object storage protocol includes a simple storage service protocol; and the distributed file storage protocol includes at least one of a network file system protocol, a common internet file system protocol, and a portable operating system interface protocol.

15. The apparatus of claim 12, wherein, The obtaining module is further configured to obtain a data read request, the data read request including directory information; if the data read request is based on the object storage protocol, the directory information includes a uniform resource locator; and if the data read request is based on the distributed file storage protocol, the directory information includes a file path. The determining module is further configured to determine a multi-level index node corresponding to a multi-level directory in the directory information. The apparatus further includes: The processing module is configured to, for each index node, read metadata of the index node based on storage location information of the index node; if the index node is not a last-level index node of a multi-level index node, determine storage location information of a next-level index node of the index node based on the metadata of the index node; if the index node is the last-level index node of the multi-level index node, determine storage location information of stored data based on the metadata of the index node, and read the stored data based on the storage location information.

16. The apparatus of any of claims 12-15, wherein, The obtaining module is further configured to, if an interface creation request based on a distributed file storage protocol is obtained, add a new index node in the target index structure based on the interface creation request; if an interface deletion request based on the distributed file storage protocol is obtained, delete an existing index node from the target index structure based on the interface deletion request; if a hard link interface request based on the distributed file storage protocol is obtained, add a new index node in the target index structure, and point the new index node to the existing index node; if a soft link interface request based on the distributed file storage protocol is obtained, add a new index node in the target index structure, and metadata of the new index node includes a string pointing to an arbitrary path; if an interface write request based on the distributed file storage protocol is obtained, write data to an offset address specified by an index node of the target index structure based on the interface write request; if an interface read request based on the distributed file storage protocol is obtained, read data from the offset address specified by the index node of the target index structure based on the interface read request.

17. The apparatus of any of claims 12-15, wherein, The obtaining module is further configured to, if an interface read request based on an object storage protocol is obtained, determine a multi-level index node based on the interface read request; and for each index node, obtain metadata of the index node. The determining module is further configured to, if the index node is a last-level index node of a multi-level index node, and it is determined based on the metadata of the index node that the interface read request is used to read a hard link file, determine storage location information of stored data based on the metadata of the index node, and read the stored data based on the storage location information; or, if it is determined based on the metadata of the index node that the interface read request is used to read a soft link file, determine path information of the soft link file based on the metadata of the index node, query a new index node according to the path information of the soft link file, determine storage location information of stored data based on metadata of the new index node, and read the stored data based on the storage location information.

18. The apparatus of any of claims 12-15, wherein, The obtaining module is further configured to, if an object storage protocol-based list object request is obtained, the list object request being used to list all objects in a storage bucket or used to filter or group all objects in the storage bucket according to a prefix and a separator, determine a multi-level index node corresponding to the storage bucket based on the list object request. For each index node, metadata of the index node is obtained, and metadata of the multi-level index node corresponding to the storage bucket is returned to the client in a specified order. In the metadata of the multi-level index node is stored, the metadata of the multi-level index node is stored in the specified order; and in the metadata of the multi-level index node corresponding to the storage bucket is obtained, the metadata of the multi-level index node corresponding to the storage bucket is obtained in the specified order. The specified order includes a dictionary order, and the dictionary order is arranged in alphabetical order.

19. The apparatus of any of claims 12-15, wherein, The obtaining module is further configured to, if a segmented upload request based on an object storage protocol is obtained, determine a multi-level index node based on the segmented upload request; and for each index node, metadata of the index node is obtained. The determining module is further configured to, if the index node is a last-level index node of the multi-level index node, determine storage location information of to-be-stored data based on the metadata of the index node. The storage module is further configured to store data blocks in the segmented upload request based on the storage location information of the to-be-stored data; wherein the to-be-stored data includes a plurality of data segments, and each data segment includes at least one data block. The storage module is further configured to, after storing the data blocks in the segmented upload request, add extension data to the metadata of the index node, the extension data including a segment identifier of a data segment to which the data block belongs, an overlay identifier of the data segment, a block identifier of the data block, and a size of the data block; wherein the overlay identifier of the data segment indicates a number of repetitions of the data segment.

20. The apparatus of claim 19, wherein, The obtaining module is further configured to, if a read request based on a distributed file storage protocol or an object storage protocol get request is obtained, determine a multi-level index node based on the read request or the get request. For each index node, metadata of the index node is obtained. The determining module is further configured to, if the index node is a last-level index node of the multi-level index node, obtain a target segment identifier of a to-be-read data segment based on the read request or the get request. Based on the metadata of the index node, a maximum overlay identifier corresponding to the target segment identifier is determined, and target extension data corresponding to the target segment identifier and the maximum overlay identifier is determined from the metadata of the index node. Based on a sorting position of the target extension data and a size of a data block in the target extension data, an offset address and a length of the to-be-read data segment are determined. Based on the metadata of the index node, storage location information corresponding to the to-be-read data segment is determined, and the to-be-read data segment is obtained based on the storage location information, the offset address, and the length.

21. The apparatus of claim 19, wherein, The acquisition module is further configured to, if the truncated operation request based on the distributed file storage protocol is acquired, determine a multi-level index node based on the truncated operation request; and acquire metadata of each index node. The determination module is further configured to, if the index node is the last level index node of the multi-level index node, determine the extension data of the plurality of data shards from the metadata of the index node, and sort the plurality of data shards based on the extension data. Based on the sorting result and the truncation size in the truncated operation request, determine the data shard to be truncated; Based on the sorting position of the extension data of the data shard to be truncated and the size of each data block in the data shard to be truncated, determine the offset address and length of the data shard to be truncated; Based on the metadata of the index node, determine the storage location information corresponding to the data shard to be truncated, and perform the emptying operation on the data shard to be truncated based on the storage location information, the offset address and the length.

22. The apparatus of any of claims 12-15, wherein The processing module is configured to, for each index node, generate a latest version identifier for the first metadata when generating the first metadata of the current version for the index node, and add the latest version identifier in the first metadata; if the index node has corresponded to the second metadata of the historical version, generate a snapshot identifier for the second metadata, the snapshot identifier being a negative value, add the snapshot identifier for the second metadata, generate a target version identifier for the second metadata, and add the target version identifier for the second metadata; The target version identifier includes a first random character, a target ordered character and a second random character, the target ordered character is obtained by performing an encryption operation on the initial ordered character and the first random character, and the initial ordered character is obtained by performing an operation on the initial ordered character of the metadata of the previous version. comprising:

23. An electronic device, comprising: a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method of any of claims 1-11. ​

Images