Improvements relating to object-based storage

Abstract

An object-based data storage system, implemented by a computer, for storing data in a plurality of objects (16A, 16B, 25A, 25B). A data storage system includes a storage medium (80) configured to store a plurality of objects. Each of the plurality of objects comprises a plurality of fields, the plurality of fields comprising a data field (18) and a separate object ID attribute field (17), the data field (18) being configured for storing data therein, the separate object ID attribute field (17) being configured for storing identification information associated with the object. The information stored within the object ID attribute field of at least one object (16A, 16B) of the plurality of objects includes metadata other than identification information associated with the at least one object. The processor is configured to access at least one object (16A, 16B) from a plurality of objects stored within the storage medium to retrieve at least information stored within an object ID attribute field (17) to retrieve metadata.

Description

[0001] This application claims priority to EP22171034.6 filed on April 29, 2022, the contents and elements of which are incorporated herein by reference for all purposes. Technical Field

[0002] The present application relates to an object-based data storage method, system and device. Background Art

[0003] File-based storage systems use a format to store and manage data as a hierarchical tree structured into a file hierarchy in which files are identifiable in the form of a directory structure. The file system stores data as a set of separate file paths. Each file path is a string that uniquely identifies a file in the form of a directory structure. These unique identifiers can include the file name, the file extension (e.g., "JPG" for JPEG files), and the path to the file. In this way, the file system controls the storage, retrieval, and display of data within the file. The extension indicates the format of the data contained in the file, for example, .txt, .png, .java, .html, .doc, etc. The directory structure defines how the file system arranges files so that users can access them.

[0004] Files and directories are identifiable in a directory structure, such as the following simple example showing the file-based concept of image and video file storage:

[0005]

[0006] Directories are unordered containers that hold files ('0001.jpg', '0002.jpg', '0001.mp4') and subdirectories ('images', 'videos', 'march-2022'). The result is a nested hierarchical system of organized files, rooted in a single top-level directory.

[0007] In contrast, object-based storage systems use a structure that manages and manipulates data stored as different units (called objects). These objects are stored in a single storage (sometimes called a "bucket") and are not arranged in files within other folders. Instead, object storage combines the data fragments that make up the file, adds all its related metadata to the file, and attaches a unique identifier to the object (UIDO). Object storage enables capabilities such as interfaces that are directly programmable by applications to access storage devices through standard object interfaces. Object storage is particularly (but not exclusively) suitable for unstructured data in which data is written once and read once or multiple times. Examples include online content, data backup, image archives, videos, pictures, and music files that can be stored as objects.

[0008] By comparison, file storage systems store data as a single piece of information in folders in a hierarchical structure to organize it among other data. When a user needs to access a file, the computer system needs a path to find it. However, instead of organizing files in a directory hierarchy, object storage systems store files in a flat organization of containers called "buckets" (e.g., in the Amazon AWS S3 system) and use unique IDs (e.g., called "keys" in the Amazon AWS S3 system) to retrieve them. Buckets are logical containers for storing objects. Users or systems can create buckets within a storage area as needed. Buckets are associated with a single compartment that can have policies that determine what actions a user can perform on a bucket and on all objects in the bucket. "Inodes" (index nodes) are data structures in Unix-type file storage systems. It is used to describe file storage system objects, such as files or directories. Each "node number" stores the attributes and disk block locations of the data of the file storage object. File storage system object attributes can include metadata (last change, access, modification time), as well as owner and permission data. However, it should be understood that although "node numbers" refer to file system "objects", this nomenclature does not mean that "node numbers" are any part of object-based storage. "Node numbers" are not any part of object-based storage, and this fact is well known in the art. A directory is a list of "node numbers" with assigned names. Therefore, the system that uses "node numbers" in the file storage system disclosed in patent application document US2016 / 283501A1 is an example of a file storage system, not an example of an object-based storage system. Examples of object-based storage can be found at: https: / / www.ibm.com / cloud / learning / object-storage.

[0009] Object storage (often referred to as object-based storage) is a data storage structure used to handle large amounts of unstructured data. Object-based storage has many fundamental differences from file storage. These include, but are not limited to, the following:

[0010] (a) Object-based storage does not use "inodes". Therefore, object-based storage does not have a hierarchical structure of folders / directories. Note that file storage systems that use "inodes" require "inodes" to point to other "inodes" in a graph structure.

[0011] (b) Object-based storage does not support in-place modification / update of object data. Instead, changes made to data in object-based storage require the entire object to be rewritten from beginning to end. Note that file storage systems that use "node numbers" require mutable block pointers within the "node numbers" that can be updated to point to new / modified data blocks.

[0012] (c) Object-based storage typically uses efficient erasure coding for storage, while file storage systems typically use RAID (“Redundant Array of Independent Disks”).

[0013] (d) Unlike file storage, object-based storage can be easily scaled to bytes of storage. Due to its very different data storage structure and more scalable organization, object-based storage typically has much higher latency than file storage. As a result of the increased efficiency, object-based storage is typically cheaper to purchase, maintain, and scale than file storage.

[0014] (e) Applications can directly access object-based storage through RESTful APIs, rather than through the operating system's file system support that requires "system programs." RESTful APIs are a structured form of application program interface (API) that uses HTTP requests to access and use data.

[0015] Any type of data can be stored as an object, regardless of the content type. Figure 1 Such an example is shown, where an object 1 typically includes the stored file data itself 3 (e.g., text, image, video, etc.), a file name (UID) 22 used to identify the object, and a quantity of metadata 2 including properties of the object created by the object storage system, such as object size, object access permissions, object creation time, etc. Each object has data (e.g., an uninterpreted sequence of bytes) and metadata (e.g., an expandable set of properties that describe the object), and is typically stored in buckets associated with other objects.

[0016] Object storage systems typically explicitly separate file metadata from data. Some distributed file systems use an object-based architecture where file metadata is stored in metadata servers and file data is stored in object storage servers. File system client software interacts with these different servers and abstracts them to present a complete object-based file system to users and applications. The command interface can include commands for creating and deleting objects, writing bytes to and reading bytes from various objects, and setting properties on and getting properties from objects.

[0017] Access to objects within an object-based file system can be managed by what is called an Access Control List (ACL). This is a list of permissions associated with an object that identifies which system processes or system users are permitted to access the object. It also specifies which operations are allowed on a given object. Each entry in the ACL can specify a subject (e.g., user #, or process #) and an operation (e.g., read, write, etc.). As a simple example, a file object may have an ACL that contains the following:

[0018] User #1: Read only.

[0019] User #2: Read, Write.

[0020] This will give user #2 permission to read and write the file, and give user #1 permission to only read the file.

[0021] Object storage systems and file storage systems have very different characteristics and are built very differently internally. Some of the differences are:

[0022] - File-based storage has a file hierarchy consisting of directories (folders) and sub-directories (sub-folders), each of which can in turn have files, whereas object storage is flat and has no actual hierarchy.

[0023] - File-based storage has a minimal amount of metadata associated with each file and directory. On a Portable Operating System Interface (POSIX) file system, this can include three types of timestamps (i.e., 'modification time', 'creation time', and 'access time') as well as user ID (UID), group ID (GID), permissions, and other attribute bits (e.g., 'symbolic link', 'directory' bit, 'setuid' bit, etc.).

[0024] - The object-based storage structure does not adhere to POSIX compatibility in its metadata and access control mechanisms (such as access control lists). It also does not adhere to Microsoft Windows software compatibility in terms of metadata and ACLs.

[0025] - Although object-based storage such as Amazon AWS S3 provides ACLs for managing access control to objects within a bucket, these ACLs are not directly compatible with ACLs supported by file storage systems such as POSIX ACLs, NFSv4 ACLs, and Microsoft Windows ACLs.

[0026] -Object-based storage can have much higher throughput scalability than file-based storage.

[0027] -Object-based storage can be easily scaled and thought of as a pool that can keep growing in size, whereas file-based storage is often limited in scaling by many orders of magnitude.

[0028] - When an application utilizes object-based storage, it is typically accessed directly by the application, for example, via a REST application programming interface (API) using the HTTP or HTTPS web protocol. Representational State Transfer (REST) ​​is a software architectural style that was created to guide the design and development of the World Wide Web architecture and is well known in the art. REST defines a set of constraints on how the architecture of Internet-scale distributed media systems (such as the World Wide Web) should behave. In contrast, when an application utilizes file-based storage, file access is typically obtained through kernel system calls, and the operating system can then directly access local disk storage, or convert to the necessary file storage network protocol.

[0029] - In general, object-based storage tends to have significantly lower storage costs per byte.

[0030] - Unlike file-based storage systems, object-based storage systems generally cannot guarantee read-after-write consistency, so that once an object is written, those writes are immediately available for other processes and nodes to read.

[0031] Neither object-native applications nor file-native applications (e.g., POSIX-compliant) can operate directly on the same data pool with the same performance level. In particular, shared access across multiple nodes is not available. Furthermore, simultaneously achieving the high throughput performance of object-based storage with the low latency performance of file-based storage is also not available. Furthermore, coherent access control across object-based and file-based interfaces is not available.

[0032] In practice, some existing solutions require copying data from an object-based representation to a file-based representation, or from a file-based representation to an object-based representation, or alternatively require running a gateway server that converts between these representations and becomes a bottleneck for performance and scalability. This replication requirement results in very poor scalability and performance, is often incompatible with POSIX file access requirements, and is limited to a single access node.

[0033] Based on the above considerations, this application is designed. Summary of the invention

[0034] The inventors have recognized that most files on a storage system tend to be created and written once, and are rarely modified even if they are frequently used (read) many times. For a given file / directory, associated attributes such as user identifier (UID), group identifier (GID), access control list (ACL) and "mode" tend to remain static and change very infrequently. Attributes (such as 'creation time') do not tend to change or change infrequently. Attributes (such as 'modification time') only change when they are actually modified, which is not the case for most files. In a given directory, there is often a lot of redundancy in UID, GID, ACL between files in the directory. Within a given directory, there are many similarities in timestamps (such as 'creation time' or 'modification time') between files in the directory. This means that attribute data tends to be highly compressible.

[0035] Unlike files in file-based storage systems, objects in object-based storage systems are generally immutable, meaning that once written, it cannot be modified. Buckets in objects in object-based storage systems cannot be nested in the same way as used in files in file-based storage systems. However, an organized structure can be achieved with proper naming conventions. For example, object-based storage of the above-noted image files and video files in one bucket can be named as follows:

[0036] Image: March-2022:0001.JPG

[0037] Image: March-2022:0002.JPG

[0038] Video: 3-2022:0001.MP4

[0039] An object-based storage system may include a 'LIST' operation (e.g., an object API query) configured to enumerate objects in a bucket. The LIST operation may support, for example, prefix-based filtering. In the LIST operation applied to the above-mentioned bucket, the objects are named with the prefix "Images:March-2022:". The LIST operation that implements this prefix-based filtering produces a list of objects consisting of images from March-2022. In this example, a colon (:) delimiter has been used. However, another delimiter such as a forward slash ( / ) may be used as a delimiter instead of a colon (:) delimiter, so that the object names in the LIST from the bucket are conceptually similar to the delimiters used in file paths in a file-based storage system. Similarly, a 'PUT' operation (e.g., an object-API query) uploads an object to an object-based storage system, while a 'GET' operation (e.g., an object-API query) retrieves an object from an object-based storage system.

[0040] The inventors have recognized that LIST operations have typically been used to retrieve a list of object names in a bucket of an object-based storage system. A LIST operation must be performed in order to provide such a list of storage objects. However, in many file-based storage systems (e.g., POSIX-compliant systems), an equivalent directory 'read' operation also requires the filling of metadata.

[0041] Rather than having an additional object API query (such as a separate GET request) to obtain the metadata required for an equivalent directory "read" operation (eg, a POSIX-compliant system), Applicants have recognized that it is possible to take advantage of such an operation that would need to be performed anyway.

[0042] The present application most generally provides a method whereby metadata in addition to the file name of the object is stored in the file name attribute field of the object in the object-based storage system. This means that a LIST operation will retrieve this storage metadata for the storage object along with (i.e., with) the file name for the object when generating a list of storage objects. This provides compatibility with file-based storage systems (e.g., POSIX-compliant systems) where an equivalent directory "read" operation needs to provide this metadata.

[0043] In a first aspect, the present application may provide a computer-implemented, object-based data storage system for storing data in a plurality of objects, the data storage system comprising:

[0044] a storage medium configured to store a plurality of objects;

[0045] wherein each of the plurality of objects includes a plurality of fields, the plurality of fields including: a data field configured to store data therein; and a separate object ID attribute field (e.g., a file name of another ID) configured to store identification information associated with the corresponding object;

[0046] wherein information stored within an object ID attribute field of at least one of the plurality of objects (e.g., an information item as described below) includes metadata in addition to identification information associated with the at least one object (e.g., the stored information may include an information item that serves as both a 'file name' and an ID for the object, and also contains bytes of information that may be interpreted as metadata in addition to simply the 'file name' itself); and

[0047] A processor is configured to access at least one object from at least one of a plurality of objects stored in a storage medium to retrieve at least information stored in a corresponding object ID attribute field thereof, thereby retrieving metadata.

[0048] References to "objects" herein may be considered to include references to encapsulation of data (e.g., an uninterpreted sequence of bytes) and metadata (e.g., an extensible set of properties that describe an object). References to "fields" herein may be considered to include references to dedicated storage areas (physical and / or logical) in a data source for containing data of a type consistent with the field type. Examples include: a data field (e.g., an uninterpreted sequence of bytes) for storing data; an attribute field for storing attributes (e.g., metadata). Preferably, an attribute field does not contain another field. References to "objects" herein may be considered to include references to discrete data units stored in a structurally flat (i.e., unstructured) data environment. References to "object-based" storage and "object-based" storage systems herein may be considered to include references to storage in which folders, directories, or complex hierarchies are not used to store / locate "objects" within a storage system. An "object" may include a unique identification (ID) number (i.e., in place of a file name and file path). The unique identification (ID) number may provide information that enables an application to locate and access an "object". An "object" may refer to a self-contained repository that may include data and / or metadata (eg, descriptive information associated with the object).

[0049] Herein, an object will be referred to as a 'metadata object' if an information item contained within an ID attribute field (e.g., a file name attribute field) of an object includes metadata associated with at least one (preferably multiple) objects that are different from the object in question. In this way, a 'metadata object' can be used as a source of metadata information related to another object or objects.

[0050] An information item contained in an ID attribute field (e.g., a filename attribute field) of an object may serve the function of a name of the object (i.e., the information conveyed by the information item as a whole is the 'filename' of the object), and the information item itself may contain therein additional information in the form of a metadata item (e.g., in encoded and / or compressed form). The additional information may include the entire information item or at least a portion of the information item. The information item contained in the filename attribute field of the object may be obtained by a query of the object-based storage system (e.g., a LIST operation that returns the contents of the filename attribute field), and the additional information (i.e., the metadata item) may then be extracted (e.g., decoded and / or decompressed, if necessary) from the information item obtained from the filename attribute field. In this way, the information item may take the form of a sequence of bytes that serves two functions: the first function is an uninterpreted sequence of bytes representing a 'metadata object' of the object in question; the second function is a vehicle for conveying an interpretable sequence of bytes representing metadata, although metadata is, of course, more than just (i.e., more than) the 'filename' of the object in question.

[0051] In other words, by virtue of being located in the filename attribute field of an object, the object-based storage system automatically accepts the information item within the filename attribute field as a service function of the filename of the object in question. In accordance with the present application, the information item may be prepared in any suitable manner so as to contain the desired information metadata item as at least a portion (e.g., in encoded and / or compressed form) of the entire information item to be placed in the filename attribute field of an object.

[0052] The information items including metadata items placed in the ID (eg, 'file name') attribute field of the object may include the following information:

[0053] (1) metadata items, including metadata associated with a given file or files;

[0054] (2) At least one ID of the file to which the metadata relates, such as a file name, or a file path of a file, or multiple files to which the metadata relates.

[0055] Preferably, the information item and metadata within the information item may include information associated with a given object. This may include information about the object itself and / or may include information about data stored in a data field of a given object. The data may include one or more files. The one or more objects to which the information item relates may be the object containing the information item, or more preferably, may be one or more objects outside the object containing the information item (e.g., another one or more separate objects). The associated information contained in the metadata within the information item may include any one or more of the following: file name; file path for the file; file identification information for the file; timestamp (e.g., creation time, modification time, or access time); user ID ('UID'); group ID ('GID'); access permissions (e.g., access control information); one or more file attribute bits. A file attribute is a piece of information associated with a file or directory that includes additional data about the file itself or its contents. For example, a byte may store attributes of a file. Each specific attribute may be assigned to a specific bit of the byte. To enable a certain attribute, the system may assign a bit value of (e.g.) 1 ('1') to the corresponding bit, the corresponding bit representing the 'on' state of the attribute. The attribute bits may correspond to one or more of the following attributes: executable; symbolic link; directory bit; setuid bit; setgid bit.

[0056] For example, as a simple illustration to aid understanding, an information item contained in a filename attribute field of a given object may include a file path appended by a metadata item, such as "Images / March-2022 / 0001.JPG", which includes metadata bytes associated with the file "0001.JPG":

[0057] ...Images / March-2022 / 0001.JPG / <metadata item>

[0058] A file path (e.g., Images / March-2022 / 0001.JPG) represents a path to a file (e.g., "0001.JPG") via a directory (e.g., "Images") and, optionally, via one or more subdirectories (e.g., "March-2022") as desired.

[0059] Thus, in response to a query of the contents of an object-based storage system (i.e., all objects) that returns the contents of the filename attribute field of the objects therein, the information returned may include:

[0060] (1) The content of the file name attribute field of the exemplary object includes: file path: images / March-2022 / 0001.JPG, and the<metadata item> ;as well as,

[0061] (2) The contents of the file name attribute field of other objects stored in the object-based storage system, including: other file paths, file names, and metadata items.

[0062] Alternatively, for example, the information item may comprise a file name without an associated file path, such as "0001.JPG", appended with a metadata item comprising metadata bytes associated with the file "0001.JPG".

[0063] The information item may combine multiple different information items into a composite information item. For example, the information item may include a composite information item including multiple additional component information items, wherein each component information item includes a file name, preferably within a related file path, appended by a corresponding metadata item including metadata bytes related to the file in question. Each of the multiple additional component information items may correspond to a respective one of a plurality of objects within the object-based storage system. For example, a first component information item may include: images / March-2022 / 0001.JPG / <metadata item1> , the second component information item may include: images / March-2022 / 0002.JPG / <metadata item2> , and the third component information item may include: Videos / March-2022 / 0001.MP4 / <metadata item3> By appending the second component information item to the first component information item, and appending the third component information item to the second component information item, the composite information item may include:

[0064] …Images / March-2022 / 0001.JPG / <metadata item1> / Images / March-2022 / 0002.JPG / <metadata item2> / Videos / March-2022 / 0001.MP4 / <metadata item3> ……wait.

[0065] Located within each component information item, and within the composite information item, there may reside a hash (e.g., a cryptographic hash) of metadata contained within the metadata item associated with the component information item (i.e., metadata associated with a given file identified by a file name and / or associated file path within the component information item). Within a given component information item, the hash of the metadata within the metadata item may be separated / isolated from the (unhashed) metadata item by the file name and / or associated file path within the component information item. That is, the component information item may include the file name and / or associated file path information sandwiched between bytes of the metadata item and bytes of the hash of the metadata item. The hash of the metadata item may be located at the end of the component information item so as to include the first byte in the byte string of the component information item. The object-based file storage system may be configured to generate information items (e.g., component information items or composite information items) according to the structure. For example, as a simple illustration to aid understanding, a composite information item may include:

[0066] … [Hash of<metadata item1> ] / Images / March-2022 / 0001.JPG / <metadataitem1> / [Hash of<metadata item2> ] / Images / March-2022 / 0002.JPG / <metadata item2> / [Hash of<metadata item 3> ] / Videos / March-2022 / 0001.MP4 / <metadata item3> wait.

[0067] Thus, metadata items and hashes of the metadata items may be used within the structure of component information items to identify the termination (beginning and end) of a given component information item within a composite information item.

[0068] A (cryptographic) hash function can be used to generate a hash (i.e., a number) from a file name or a full path to a file name (including a file name) within a component information item. A hash function is a method of generating random numbers. Here, a reference to a hash generated by applying a hash function to something can be replaced by a reference to a random number (e.g., for associating with something), the random number being generated by a means other than applying a hash to something. The hash can be up to 128 or 256 bits long, and it is extremely unlikely that two files collide (i.e., have the same hash). A hash can be a one-way function, meaning that a metadata item or a file path or file name cannot generally be reconstructed from its hash. However, if you have a list of file names and / or file paths, you can recalculate the hash of each file name and / or file path and match that hash with the hash in the retrieved composite information to identify which one corresponds to the file name and / or file path.

[0069] Within each information item, such as within each component information item within a composite information item, may reside a hash (e.g., a cryptographic hash) of the file name and / or file path contained within the information item. Within a given information item, a hash of the file name and / or file path may be provided in place of the (unhashed) file name and / or associated file path within the information item. That is, an information item may include information identifying the file name, and / or associated file path information in hashed form only. An object-based file storage system may be configured to generate information items (e.g., unique information items or component information items) according to this structure. For example, as a simple illustration to aid understanding, a composite information item may include:

[0070] In the personal or structured information item:

[0071] …[Hash of< / Images / March-2022 / 0001.JPG / > ] / <metadata item1>

[0072] In a composite information item:

[0073] …[Hash of< / Images / March-2022 / 0001.JPG / > ]<metadata item1> / [Hash of<Images / March-2022 / 0002.JPG / > ]<metadata item2> / [Hash of< / Videos / March-2022 / 0001.MP4 / > ]<metadata item3> wait.

[0074] The object-based file storage system can be configured to decode the retrieved information item by selecting an object of interest within the object-based storage system and selecting a file name and / or file path of a file stored within the selected object, and by applying the same hash function used to generate a hash of the file name and / or file path within the metadata object to the selected file name and / or file path to generate a comparison hash. The object-based file storage system can be configured to compare the comparison hash with the hash of the file name and / or file path within the metadata object, and if the comparison hash is found to be the same as the hash of the file name and / or file path within the information item containing the metadata item, the selected file name and / or file path is identified as a metadata item within the information item corresponding to the metadata object. Using a hash of the file name and / or file path within the information item helps to reduce the storage space required to store information identifying the file name and / or file path. Of course, if storage space is available to do so, the information identifying the file name and / or file path can simply include the file name and / or file path in a non-hashed form. In this way, metadata associated with a file stored within an object of interest can be identified and retrieved from the metadata object.

[0075] As a simple example for understanding, an example of a single information item of / images / March-2022 / 0001.JPG could be:

[0076] / Images / March-2022 / .meta / [Full hash of / Images / March-2022 / 001.JPG][part number(1 / 1)][timestamp][payload]

[0077] The portion of the information item " / Images / March-2022" is an example of what is known in the art as a "prefix" of a file path. The "prefix" portion of a file path corresponds to the portion of the file path up to, but not including, the file name of the file to which the file path relates. The file name will be found at the end of the file path. In this sense, the "prefix" of a file path can be thought of as a truncation of the file path in which the file name has been removed or does not exist. In this example, the full file path is " / Images / March-2022 / 001.JPG", and this is the file path for the file "001.JPG", so the "prefix" of the file path for his file is " / Images / March-2022". The use of a prefix has the advantage that it will be revealed in the content listing produced by performing a LIST operation, thereby making the interpretation of the content listing easier and more efficient. Therefore, desirably, in a preferred embodiment, the information item includes the prefix portion of the file path.

[0078] The portion of the information item " / .meta" is an optional portion of the information item and corresponds to an example of a Unicode symbol which may be included if desired to help identify the source or origin of the information item. If desired, this may be appended to the file path prefix, as shown in this example. Thus, desirably, in a preferred embodiment, the information item includes the Unicode symbol. This may be appended to the prefix portion of the file path, if present.

[0079] Appended to the file path prefix (or to the Unicode symbol, if present) is a hash (e.g., a cryptographic hash) of the full file path. In this simple example, the appended hash is the hash of the file path " / Images / March-2022 / 001.JPG". Thus, desirably, in a preferred embodiment, the information item includes the hash of the file path of the file appended to the prefix portion of the file path. Optionally, the hash of the file path may be appended to the prefix portion of the file path via the intermediate Unicode symbol (if present).

[0080] The information item may include a payload (e.g., a metadata item) that includes a bitmap configured to identify the type of metadata contained within the payload. For example, "[payload]" in this simple example may be, for example:

[0081] [bitmask][metadata1][metadata2]

[0082] The payload may be compressed. The "[bitmask]" item may be a bit mask corresponding to or identifying which type of information is conveyed by the metadata contained in the payload. For example, the bit mask may be an ordered sequence of n bits (e.g., n=5), where the position of the bit within the sequence identifies the type of metadata (information type), and the value of the bit identifies whether metadata of that type is present within the payload (e.g., within the metadata appended to the bit mask). The ordering of different types of metadata within the payload corresponds to the ordering of the bits within the bitmap. As a simple illustration, a bit mask value of [bitmask=01100] indicates that the two pieces of metadata "[metadata1][metadata2]" appended to the bit mask include possible five pieces of metadata. This is indicated by the presence of two bit values ​​"1" and three bit values ​​"0". The position of the first bit value "1" indicates that the first metadata corresponds to a group ID ("GID"). The position of the second bit value "1" indicates that the first metadata corresponds to a modification time ("mtime") value:

[0083] [metadata1][metadata2]=[gid][mtime]

[0084] Therefore, as a simple illustrative example, if an object-based storage system performs a LIST operation to list the content stored corresponding to the file path prefix " / Images / March-2022", the result is as follows:

[0085] / Images / March-2022 / .meta / ajkshkajshdkla[1][payload part1]

[0086] / Images / March-2022 / .meta / fkjsdfkjhasfsv[2][payload part2]

[0087] / Images / March-2022 / .meta / ajkshkajshdkla[3][payload part3]

[0088] / Images / March-2022 / .meta / ajkshkajshdkla[2][payload part2]

[0089] / Images / March-2022 / .meta / fkjsdfkjhasfsv[1][payload part1]

[0090] The object-based storage system can be configured to identify common hashes in the listed content generated by implementing the LIST operation (e.g., "fkjsdfkjhasfsv", which is the full hash of / Images / March-2022 / 001.jpg). In this simple example, the object-based storage system can be configured to identify common hashes in each of two of the following five listed contents (information items):

[0091] / Images / March-2022 / .meta / fkjsdfkjhasfsv[1][payload part1]

[0092] / Images / March-2022 / .meta / fkjsdfkjhasfsv[2][payload part2]

[0093] The identified part numbers [payload part 1] and [payload part 2] identify the first of the two listed items as the first part of a larger payload, and the second of the two listed items as the second part of a larger payload. Thus, desirably, in a preferred embodiment, the information item includes a part number identifying that the payload is a component of a larger payload that has been divided into multiple parts and / or identifying which component of the larger payload is contained in the payload (i.e., provided by the payload). This may be appended to the prefix portion of the file path, if present.

[0094] The object-based storage system may be configured to reassemble multiple partial payload parts retrieved from an information item. Here, in a simple example, the object-based storage system may be configured to reassemble two payload parts into one payload:

[0095] [payload]=[payload part1]&[payload part2] combination.

[0096] The object-based storage system may be configured to identify a common hash (e.g., "ajkshkajshdkla," which is the full hash of / Images / March-2022 / 002.jpg) among multiple information items listed as a result of a LIST operation. The object-based storage system may be configured to identify associated payloads (e.g., metadata items) of multiple information items that carry a common hash as containing component payload portions that may be combined to form a larger payload. The object-based storage system may be configured to combine associated payloads (e.g., metadata items) of multiple information items that carry a common hash to form a larger payload. In this simple example, the results of a LIST operation may include:

[0097] / Images / March-2022 / .meta / ajkshkajshdkla[1][payload part1]

[0098] / Images / March-2022 / .meta / ajkshkajshdkla[2][payload part2]

[0099] / Images / March-2022 / .meta / ajkshkajshdkla[3][payload part3]

[0100] The identified part numbers [payload part 1], [payload part 2], and [payload part 3] identify the three listed items as the first, second, and third parts of a larger payload. The object-based storage system can be configured to reassemble the three payload parts into one payload: [payload] = [payload part 1] & [payload part 2] & [payload part 3] combination.

[0101] As a simple but illustrative example, an example of a merge of information items for three file paths: / Images / March-2022 / 0001.JPG, / Images / March-2022 / 0002.JPG, and / Images / March-2022 / 003.JPG may be as follows:

[0102] / Images / March-2022 / .meta / [Hash of payload][part number(1 / 3)][part ofpayload split over parts]

[0103] / Images / March-2022 / .meta / [Hash of payload][part number(2 / 3)][part ofpayload split over parts]

[0104] / Images / March-2022 / .meta / [Hash of payload][part number(3 / 3)][part ofpayload split over parts]

[0105] Here, the payload has a size such that it is divided into three information items, which together contain the combined payload of the information items divided into three parts. In another example, the payload may have a size such that it does not need to be split over multiple information items in this way. In this case, there will be only one part number (e.g., "[Part Number (1 / 1)]").

[0106] Notably, the encoding of the merged information item differs in that it has been appended to the file path prefix (or to the optional Unicode symbol / .Meta / , if present), and a hash of the entire payload is split across multiple information items (e.g., "[hash of payload]"), as opposed to a hash of the file path (e.g., " / Images / March-2022 / 001.JPG]"), which is used in one unmerged information item discussed above. Thus, desirably, in a preferred embodiment, the information item includes a hash of the entire payload, where the entire payload is split across multiple information items. The object-based storage system can be configured to generate (and / or interpret) the information item accordingly.

[0107] In particular, the "[Hash of payload]" contained in the information item in question need not correspond to the hash of any one "[part of payload split over parts]", but rather, the "[Hash of payload]" preferably corresponds to the hash of the entire payload of which each "[part of payload split over parts]" forms a part. In other words, each "[part of payload split over parts]" can be combined together into a larger original (unsplit) payload, and the "[Hash of payload]" corresponds to the hash value of the larger original (unsplit) payload. The object-based storage system can be configured to split the larger original payload into its parts and combine the split payload parts upon subsequent retrieval.

[0108] This hashing of the larger original (unsplit) payload allows an object-based storage system to identify multiple information items that share the same hash as being associated with the same split payload (e.g., the three information items shown above would have the same "[Hash of payload]" value).

[0109] The hash of the payload may in turn be appended by a part number identifying a specified one of multiple ordered parts of the payload in question (e.g., "[part number (1 / 3)]", "[part number (2 / 3)]", "[part number (3 / 3)]"). The part number may then be appended by the payload. An object-based storage system may be configured to read and interpret the part numbers and identify the payload appended thereto as a specified part within a sequential set of a specified number of parts that may be combined together into a larger payload. An object-based storage system may be configured to combine portions of a separate payload according to an order indicated by the part numbers.

[0110] The object-based storage system can be configured to read and interpret hashes of payloads appearing within a consolidated information item (e.g., "[Hash of payload]") as a means of identifying other consolidated information items of the object-based storage system that contain different portions of the payload that are to be reassembled into a reconstructed payload when retrieved. The object-based storage system can be configured to read and interpret payload part numbers (e.g., "[Part Number (1 / 3)]") accordingly to indicate the order of the components of the payload and the sequence with which those payload parts should be reassembled when reconstructing the entire payload.

[0111] Thus, as an illustrative example, if an object-based storage system performs a LIST operation to list the content stored corresponding to the file path prefix " / Images / March-2022", the result is as follows:

[0112] / Images / March-2022 / .meta / abkjhktjshdkla[1 / 3][payload part1]

[0113] / Images / March-2022 / .meta / fkjrajljhasfsv[1 / 2][payload part1]

[0114] / Images / March-2022 / .meta / abkjhktjshdkla[2 / 3][payload part2]

[0115] / Images / March-2022 / .meta / abkjhktjshdkla[3 / 3][payload part3]

[0116] ...wait.

[0117] Here, "[Hash of payload]" is "abkjhktjshdkla", which identifies that those listed entries that share this hash have a partial payload that corresponds to one larger payload split over three parts. "[Hash of payload]" identified as "fkjrajjhasfsv" does not correspond to this larger payload, but rather to another larger payload.

[0118] As mentioned above, the [payload] may include different metadata and corresponding bitmasks. For example, the payload may include:

[0119] [bitmask][metadata1][metadata2][metadata3]...etc.

[0120] The merged information item contains a composite information item containing information derived from the multiple component information items involved in the merge process. Therefore, the payload may also include a hash of the file paths associated with each component information item merged therein. This may be in the form of a list. A simple example is:

[0121] [ / Images / March-2022 / 001.JPG][ / Images / March-2022 / 002.JPG][ / Images / March-2022 / 003.JPG]

[0122] Alternatively, or in addition, the object-based file storage system may be configured to decode the retrieved compound information item using a hash of the metadata within the compound information item.

[0123] The examples of preferred structures for information items described above are not limiting, and it should be understood that other structures for information items may be implemented. The inventors have found that the preferred structures for information items described above are particularly effective in practice and allow for fast information retrieval with efficient use of hardware resources within an object-based storage system. However, as examples of some alternative ways to construct information items, and the use of such alternative structures, the following examples are provided.

[0124] For example, the object-based file storage system may be configured to decode the retrieved compound information item by selecting a hash of a metadata item within the compound information item, by selecting the metadata item within the compound information item, and by applying to the selected metadata item (i.e., in its original unhashed form) the same hash function that was used to generate the selected hash of the metadata item to generate the comparison hash. The object-based file storage system may be configured to compare the comparison hash with the selected hash, and if the comparison hash is found to be the same as the selected hash of the metadata item, then identifying the selected metadata item as corresponding to the selected hash.

[0125] The file name and / or file path located between the selected metadata item identified in this manner and the selected (identical) hash of the metadata item can then be identified as the file name of the file and / or the file path of the file associated with the component information item. In other words, within the composite information item, the identified metadata item and its associated hash are located as the file name of the "end of book" file and / or the file path of the file associated with the metadata. This means that, for each of a plurality of files, the file name of each file and / or the file path of each file and its associated metadata can be readily identified and retrieved from the composite information item, which itself can be retrieved from the contents of the file name attribute field of just one object within the object-based storage system.

[0126] This functionality is particularly useful in situations where a composite information item is too long to be contained in just one metadata item. In this case, the composite information item can be split into multiple parts, and each of the multiple parts can be stored in an ID attribute field (e.g., a file name attribute field) of a corresponding one of a plurality of metadata objects. The entire composite information item can then be collectively retrieved from the contents of the file name attribute field of all metadata objects within the object-based storage system. As a simple example for understanding, consider the following composite information item split between two separate metadata objects (metadata object #1 and metadata object #2):

[0127] … [Hash of<metadata item1> ] / Images / March-2022 / 0001.JPG / <metadataitem1> / [Hash of<metadata item2> ] / Images / March-2022 / 0002.JPG / <metadata item2> / [Hash of<metadata item 3> ] / Videos / March-2022 / 0001.MP4 / <metadata item3> …etc., etc.

[0128] The information item will be split at the following locations within the component information items:

[0129] [Hash of<metadata item 3> ] / Videos / March-2022 / 0001.MP4 / <metadata item3>

[0130] The resulting two separate metadata objects (Metadata Object #1 and Metadata Object #2) contain the following information items in their respective filename attribute fields:

[0131] Information items in metadata object #1:

[0132] … [Hash of<metadata item1> ] / Images / March-2022 / 0001.JPG / <metadataitem1> / [Hash of<metadata item2> ] / Images / March-2022 / 0002.JPG / <metadata item2> / [Hash of<metadata item 3> ] / Videos / …

[0133] Information items in metadata object #1:

[0134] …March-2022 / 0001.MP4 / <metadata item3> ...wait.

[0135] The object-based file storage system may be configured to generate a "<metadata item3> " and compares the comparison hash to a hash selected from: [Hash of<metadata item1> ]; [Hash of<metadata item2> ]; [Hash of<metadata item3> ]. Object-based file storage systems can store selected metadata items.<metadata item1> " is identified as corresponding to a hash selected from: [Hash of<metadata item1> ]; [Hash of<metadata item2> ]; [Hash of<metadata item3> ]. If the compared hash is found to be the same as the selected hash. The "<metadata item3> " can be identified as " within the information item stored in metadata object #1<metadata item3> " corresponds to. Then, in "<metadata item 3> "and"<metadata item 3> "Hash of<metadata item 3> ]” can be identified as the file path of the file “0001.MP4” associated with the component information item.

[0136] The object-based file storage system may be configured to store a cryptographic hash function for generating a hash of a metadata item, and may be configured to generate a hash of the metadata item within the retrieved composite information item using the stored cryptographic hash function. The recovered metadata item may then be used to identify the location of a corresponding (identical) hash within the retrieved composite information item, and thereby identify the location of a corresponding file name and / or file path of a file associated with the metadata item.

[0137] Within a composite information item, a metadata item associated with a file name and / or file path may be appended to the file name and / or file path. For example, within a composite information item, a file name and / or file path may be appended to a hash of a metadata item associated with a file name and / or file path. When both conditions are met, the file name and / or file path may thus be sandwiched between a metadata item and the hash of the metadata item.

[0138] The location of the metadata item and the hash of the metadata item within the retrieved composite information item is known, thereby revealing the location of the file name and / or file path associated with the metadata item. The object-based file storage system can be configured to obtain the location of both the metadata item and the hash of the metadata item within the retrieved composite information item. The object-based file storage system can be configured to retrieve the file name and / or file path associated with the metadata item from a location between the metadata item (e.g., its terminal) and the hash of the metadata item (e.g., its terminal) in the retrieved composite information item.

[0139] The compound information item may be configured such that each file name and / or file path, each metadata item, and each hash of the metadata item is delimited from other parts of the compound information item by a separator symbol. If each file name and / or file path may be delimited from the metadata item by a separator symbol, and may be delimited from the hash of the metadata item by a separator symbol. For example, each hash of the metadata item may be delimited from the file name and / or file path by a separator, and may be delimited from the metadata item by a separator. For example, each metadata item may be delimited from the file name and / or file path by a separator, and may be delimited from the hash of the metadata item by a separator.

[0140] The compound information item may be configured such that the delimiter is present at least once in every 255 characters of the compound information item (i.e., the delimiter symbol may occur more regularly than once in every 255 characters, but preferably not less frequently than this). This helps to improve compatibility with information formats employed in various applications running on object-based file storage systems. It has been found that by enforcing the path delimiter every 255 characters, this is effective in ensuring compatibility with different object storage systems. That is, this ensures that a bucket full of objects can be moved to a different object storage system without incompatibility.

[0141] The separator symbol may be a "slash" symbol (ie, " / " or "\"), or a colon symbol (ie, ":") or other suitable and appropriate symbols, which will be apparent to those skilled in the art.

[0142] Restricting the character sets used in compound information items also helps improve compatibility.

[0143] Information including hashes of metadata items may be encoded as cryptographic hashes having a length of up to 128 bits, but preferably less, such as between 30 and 60 bits (e.g., 32 or 40 bits). Information including file paths (or file names) may be encoded as cryptographic hashes, for example, having a length of up to 128 bits, but preferably less, such as between 32 and 60 bits (e.g., 32 or 40 bits).

[0144] Storing a 128-bit or 256-bit hash may be more bits than would be needed if one wanted to uniquely identify a file from only a hundred files, or even from up to tens of thousands of files. Therefore, a narrower (shorter) hash (i.e., truncated, i.e., fewer bits) may be used for the hash. For example, 32 bits may be able to uniquely identify thousands of file names, and one way to generate 32 bits is to truncate a 128-bit hash to a 32-bit hash. Such truncation may be accomplished by throwing out the top and / or bottom bits of the 128-bit hash, i.e., for example. The composite information item within the object may then list many such 32-bit hashes together, and encode metadata to correctly map corresponding metadata to each of them.

[0145] However, there is always a small chance that a new object may be created in an object-based file storage system that has a file name (or path name) that has a 32-bit hash that is exactly the same as a file name (or path name) that has been generated from another existing file name (or path name) of an existing object in the object-based file storage system. In this case, the 32-bit hash that has been encoded in the composite information item no longer uniquely identifies the original object that it was originally intended to identify (i.e., the hash of the file name or file path of the original object in question is no longer a unique hash). This is an example of a "hash collision".

[0146] Preferably, in accordance with the present application, this can be addressed by applying timestamp-based filtering. For example, by providing each information item (or each component information item within a composite information item) with a retrievable timestamp that is configured to identify the time at which the object-based storage system created / added the object associated with the information item, the retrieved timestamps can be compared to determine whether any one object in the object-based storage system has a creation / addition time before or after the creation / addition time of any other such object. For example, if a "hash collision" occurs, the object-based storage system can be configured to use the timestamp information as an additional piece of information that is utilized to distinguish between two files having the same hash.

[0147] Preferably, in accordance with the present application, a wider hash (e.g., an untruncated hash, such as 128 bits wide or 256 bits wide) may be stored in the data field of an object that contains a truncated version of the hash within the composite information item stored within the filename attribute field of the same object. This serves as a fallback provision in the event that the truncation has been subject to a hash collision. For example, if objects are copied to new buckets, the object timestamp information for the new buckets of those copied objects may be unreliable, and the wider hash stored in the data field of the copied objects within the new bucket serves as a fallback that may be used if a hash collision occurs for the truncated hash.

[0148] merge

[0149] The object-based file storage system can be configured to generate metadata objects by generating new metadata items or by overwriting existing metadata items. This process is referred to as "merging" here. It is the process of providing metadata objects to serve multiple objects in the object-based file storage system. The multiple objects in question may include all objects in the object-based file storage system, or all objects in the storage bucket of the object-based file storage system. This is referred to as "full merging" here. Alternatively, the multiple objects in question may include some (but not all) objects in the object-based file storage system, or some (but not all) objects in the storage bucket of the object-based file storage system. This is referred to as "partial merging" here.

[0150] A metadata object may be generated by an object-based file storage system as a new metadata object to serve multiple new objects that are newly (e.g., simultaneously) added to the object-based file storage system. In this case, the new metadata object and the multiple new objects it serves are stored together with existing objects (possibly including other metadata objects) that already exist in the object-based file storage system (e.g., in the same storage bucket). This is an example of a "partial merge".

[0151] Alternatively, a fully merged metadata object may be generated by the object-based file storage system, either within a specified portion of the object-based file storage system (e.g., within the same storage bucket), by overwriting an existing metadata item, or by generating a new metadata item. The resulting fully merged metadata item thus generated thereafter serves all objects within the object-based file storage system or within the specified portion of the object-based file storage system, including any new objects newly added (e.g., simultaneously) with existing objects that already exist. A full merge may be implemented by the object-based file storage system according to the following method:

[0152] Step A: Obtaining ID information (e.g., file name) associated with a first (e.g., pre-existing) metadata item. This will include an information item that includes a metadata item as described above;

[0153] Step B: Obtaining ID information (e.g., file name) associated with a second (e.g., newly generated) metadata item. This will include an information item that includes the metadata item as described above;

[0154] Step C: decoding the ID information (e.g., file name) obtained in step A and step B to obtain the file name and / or file path and metadata stored in the ID information;

[0155] Step D: re-encoding the file name and / or file path and metadata obtained from step C into a composite information item (e.g., as described above);

[0156] Step E: Generate a metadata object containing the composite information item generated by step D in its ID attribute field (e.g., file name attribute field). This can be achieved by generating a new metadata object, or by overwriting one of the first and second metadata objects, optionally deleting the other. The result is a fully merged metadata item.

[0157] When generating a fully merged metadata item, the object-based file storage system can be configured to apply any one or more of the above-described methods relating to the use of hashes, application to metadata items and / or file names and file paths, splitting of information items across two metadata objects (if necessary), and timestamp filtering to avoid hash collisions as described above.

[0158] Typically, objects within an object-based storage system are generally considered to be immutable and can only be overwritten, or new objects created, but cannot be renamed. The processor is preferably configured to access a selected object from a plurality of objects stored in a storage medium to store the metadata (e.g., an information item containing a metadata item) in its object ID attribute field. Thus, the processor can overwrite the object. Alternatively, or in addition, the processor is preferably configured to generate an object containing the metadata (e.g., an information item containing a metadata item) in its object ID attribute field for storage in a plurality of objects stored in the storage medium for storage. Thus, the processor can create a new object.

[0159] Preferably, the information (e.g., the information item) stored in the object ID attribute field of the at least one object (e.g., the information item containing the metadata item) includes metadata associated with at least one other object from the plurality of objects, the at least one other object being different from the at least one object. Thus, the object may be a 'metadata object' as described above.

[0160] Ideally, the information (e.g., one or more information items) stored in the object ID attribute field of the at least one object (e.g., an information item containing a file name or file path, or a hash thereof) includes identification information associated with at least one other object of the plurality of objects other than the at least one object. For example, the object may be a 'metadata object' as described above, which includes an information item containing a file name or file path, or a hash thereof, and a metadata item.

[0161] Preferably, the metadata (e.g., metadata item) includes information associated with data stored in a data field of an object in the plurality of objects, the plurality of objects including one or more of the following: file name; file path; file identification information of the file; timestamp (e.g., creation time, modification time, or access time); user ID ('UID'); group ID ('GID'), access permissions (e.g., access control information); one or more file attribute bits. The attribute bits may correspond to one or more of the following attributes: executable; symbolic link; directory bit; setuid bit; setgid bit.

[0162] Ideally, multiple objects including the at least one object are arranged in a common storage bucket, wherein the information (e.g., information item) stored in the object ID attribute field (e.g., file name attribute field) of the at least one object includes metadata (e.g., metadata item) and / or identification information associated with at least one other object among the multiple objects arranged in the common storage bucket, the at least one other object being different from the corresponding object. The information (e.g., information item) stored in the object ID attribute field of the at least one object may include metadata (e.g., metadata item) associated with the multiple objects arranged in the common storage bucket. For objects in the common storage bucket, the object may be a 'metadata object' as described above.

[0163] The object data field of the at least one object is preferably an empty field containing no data (i.e., zero bytes). Alternatively, the object data field of the at least one object may contain information about ID attributes (e.g., file names or file paths) of one or more files to which the information item in the ID attribute field of the object relates. If the object is, for example, a 'metadata object' as described above, the data field of the metadata object may contain a hash of a file name or file path that is larger (e.g., not truncated at 128 bits or 256 bits) than a truncated / shorter hash of the same file name or file path contained in the information item in the ID attribute field of the metadata object.

[0164] Metadata may include metadata associated only with at least one object. For example, an object may not be a 'metadata object' such that the information item contained in the ID attribute field of the object contains metadata relating to files within the object itself, but not metadata within other objects.

[0165] The information stored in the object ID attribute field of each of the plurality of objects may include metadata in addition to the identification information associated with the respective object. That is, each of the plurality of objects may be a 'metadata object' giving a plurality of 'metadata objects' within the object-based storage system.

[0166] The object ID attribute field may include a file name attribute field or a unique identifier (UIDO) field, and the identification information associated with the corresponding object includes a file name, or a file path associated with the corresponding object (e.g., including a file name) or a unique identifier (UIDO).

[0167] Preferably, the metadata is stored in a compressed form. For example, an information item containing an ID attribute field of an object may contain a metadata item that includes metadata stored in a compressed form. Alternatively, or in addition, illustratively, an information item containing an ID attribute field of an object may contain a metadata item that includes a hash (e.g., a cryptographic hash) of the metadata.

[0168] The identification information associated with each object may include a hash of a unique identifier (UIDO) associated with the object, or a hash of a unique identifier (UIDO) associated with a file stored within the object, the unique identifier (UIDO) being present in multiple objects and containing the metadata. For example, an information item contained in an ID attribute field of an object may include a hash of a file name or a file path (e.g., a cryptographic hash).

[0169] The hash may encode the path of a file name or file path associated with one of the multiple objects. The file name or file path may be associated with a file stored within another object (e.g., within a data field of another object). In this sense, the path of the file name or file path in question is associated with the object by extension.

[0170] The aforementioned identification information associated with the corresponding object may include a hash of a file path associated with one of the multiple objects. The aforementioned identification information associated with the corresponding object may include a hash of one or more of the following: a file name, a file path, or file identification information associated with one of the multiple objects and containing the metadata. The aforementioned identification information associated with the corresponding object may include at least one hash of at least one metadata item of the multiple metadata items associated with the corresponding one of the multiple files to map the metadata item to a corresponding file name associated with the object of the multiple objects.

[0171] Typically, metadata within an object's ID attribute field may include an access control list (ACL) containing a list of permissions associated with accessing files within an object stored in the plurality of objects, and the plurality of objects preferably include at least one other object containing the files to which the access control list relates.

[0172] The metadata (e.g., metadata item) or information item within the ID attribute field (e.g., file name attribute field) of the object may include information defining an access control list (ACL) containing a list of permissions associated with access to data (e.g., files) stored within an object of the plurality of objects. The information defining the access control list (ACL) may be stored within a first object (e.g., a dedicated object, e.g., an "ACL object"). The metadata (e.g., metadata item) or information item within the ID attribute field of the object may include a pre-stored hash of an access control list entry within the access control list, the access control list defining access controls applicable to data associated with the metadata within the information item (e.g., metadata item). The pre-stored hash of the access control list entry may be generated by applying a preset hash function (e.g., a cryptographic hash function) to the access control list entry. The object-based storage system may be configured to apply the preset hash function to the access control list entry, thereby generating a value thereof, and include such a selected hash value within the metadata (e.g., metadata item) or information item as a pre-stored hash within the ID attribute field of the object.

[0173] An information item including a pre-stored hash of access control list entries / information may be stored in an ID attribute field of an object that is distinct from an object containing an access control list (ACL) (e.g., distinct from an "ACL object"). To determine which access controls are applicable to a file referenced by an information item, an object-based storage system may be configured to retrieve an access control list (ACL) from the object containing the access control list (ACL) and apply a preset hash function (e.g., a cryptographic hash function) to access control list entries within the retrieved access control list to generate a corresponding comparison hash for one or more (e.g., each) corresponding access control list entries. The object-based storage system may be configured to compare the resulting comparison hash value with a pre-stored hash within a given information item from the object and identify which comparison hash matches the pre-stored hash. Access controls defined by access control list entries having a hash match (i.e., the same as the pre-stored hash) may then be identified as controls that apply to data associated with the given information item (i.e., the metadata items therein).

[0174] If the access control list comprises an ordered list of a plurality of consecutive list entries, wherein each list entry contains access control information defining access controls applicable to data (e.g., files) stored in an object of the plurality of objects. As a simple illustrative example for better understanding, an ACL for a file contained in a data field of a plurality (e.g., 1 to 'n') of separate objects may comprise the following ordered list:

[0175]

[0176] Metadata items stored within an ID attribute field (e.g., a file name attribute field) of a given object among multiple (e.g., 1 to 'n') independent objects may be associated with one or more of the data (e.g., files) to which the ACL entry relates. For example, the entry: "ACL entry #1" contains access control information that defines access controls applicable to data (e.g., files) stored within object #1 and is associated with a metadata item that references the file within object #1. Similarly, the entry: "ACL entry #2" contains access control information that defines access controls applicable to data (e.g., files) stored within object #2 and is associated with a metadata item that references the file within object #2, and so on.

[0177] The metadata item within the object may include a pre-stored hash of an access control list entry within the access control list (e.g., 'hash[ACL entry#2]'), which may be stored in an object other than the object containing information defining an access control list (ACL) (e.g., an object other than 'ACL object'). The object-based storage system may be configured to apply a pre-set hash function (e.g., a cryptographic hash function) to access control entries within the access control list.

[0178] For example, to generate a hash list:

[0179] Hash#1=Hash[ACL entry#1]

[0180] Hash#2=Hash[ACL entry#2]

[0181] …

[0182] Hash#n=Hash[ACL entry#n]

[0183] The object-based storage system may be configured to identify which hash matches a hash within a given metadata item and apply / associate the access control entry for the matching hash to the data associated with the given metadata item. For example, object #2 may include a pre-stored hash = hash[ACL entry #2], and this matches 'Hash #2', thereby identifying that ACL entry #2 contains access control restrictions applicable to the file referenced by the information item (e.g., its metadata item) containing the pre-stored hash = 'hash[ACL entry #2]'.

[0184] Thus, in general, the plurality of objects may include: at least one object including metadata, the metadata including an access control list, and at least one other object to which the access control list relates.

[0185] At least one other object may include information stored within its corresponding object ID attribute field, the information including metadata associated with the corresponding other object, the metadata containing information about one or more files or directories referenced by the access control list, and information identifying the at least one object including the metadata, the metadata including the access control list.

[0186] The metadata may include one or more symbolic links (also referred to as "Symlinks", or "SYLKs") that are configured to be interpreted and followed by the processor as a path to a file or directory. For example, a symbolic link may include a "target_path" that defines the relative or absolute path to which the symbolic link points, and a "link_path" that defines the path of the symbolic link. Preferably, the one or more symbolic links are configured to conform to a POSIX-compliant operating system. The Portable Operating System Interface (POSIX) is a series of standards for maintaining compatibility between operating systems, which is well known in the art.

[0187] In a second aspect, the present application may provide a method for object-based data storage implemented by a computer for storing data in a plurality of objects, the method comprising:

[0188] Providing a plurality of objects, wherein each of the plurality of objects comprises a plurality of fields, the plurality of fields comprising: a data field configured for storing data therein; and a separate object ID attribute field configured for storing identification information associated with the object;

[0189] wherein the information stored in the object ID attribute field of at least one object of the plurality of objects includes metadata other than identification information associated with the at least one object;

[0190] storing a plurality of objects on a storage medium;

[0191] At least one of the plurality of objects is accessed, by the processor, to retrieve at least information stored in a corresponding object ID attribute field thereof, thereby retrieving metadata.

[0192] The method may include, by a processor, accessing an object selected from a plurality of objects stored in a storage medium to store metadata in its object ID attribute field, and / or generating an object containing metadata in its object ID attribute field for storage in a plurality of objects stored in the storage medium.

[0193] The method may include storing, within information stored within an object ID attribute field of the at least one object, metadata associated with at least one other object of the plurality of objects other than the at least one object.

[0194] The method may include storing, within information stored within an object ID attribute field of the at least one object, identification information associated with at least one other object of the plurality of objects other than the at least one object.

[0195] Metadata may include information associated with data stored in a data field of an object of a plurality of objects, the plurality of objects including one or more of: a file name; a file path; file identification information.

[0196] The method may include arranging multiple objects including at least one object in a common storage bucket, wherein information stored in an object ID attribute field of the at least one object includes metadata and / or identification information, and the metadata and / or identification information is associated with at least one other object among the multiple objects arranged in the common storage bucket that is different from the corresponding object.

[0197] In the method, the information stored in the object ID attribute field of the at least one object may include metadata associated with a plurality of objects arranged within a common bucket.

[0198] The object data field of at least one object may be an empty field (ie, zero bytes) containing no data.

[0199] According to the method, the metadata may include metadata objects associated only with the at least one.

[0200] In the method, the information stored in the object ID attribute field of each of the plurality of objects may include metadata other than the identification information associated with the corresponding object.

[0201] In the method, the object ID attribute field may be a file name attribute field or a unique identifier (UID) field, and the identification information associated with the corresponding object includes a file name or a unique identifier (UID) associated with the corresponding object.

[0202] The method may include storing the metadata in compressed form.

[0203] In the method, the identification information associated with the corresponding object may include a hash, the hash including a unique identifier (UID) associated with an object in the plurality of objects and including the metadata. The hash may encode a path of a file name associated with one of the plurality of objects. The hash may encode a plurality of metadata items associated with the plurality of corresponding files into a common hash code configured to map each metadata item to a corresponding file name associated with one of the plurality of objects.

[0204] In the method, the metadata may include an access control list (ACL) containing a list of permissions associated with access to an object of the plurality of objects.

[0205] According to the method, the plurality of objects may include: at least one object including metadata including an access control list; and at least one other object to which the access control list relates.

[0206] The metadata may include one or more symbolic links that are configured to be interpreted and used after the processor as a path to a file or directory.

[0207] In a third aspect, the present application may provide a data processing device, comprising a processor configured to execute the above method.

[0208] In a fourth aspect, the present application may provide a computer-readable medium having instructions stored thereon, which, when executed by a computer, causes the computer to perform the steps of the method according to the above method.

[0209] In a fifth aspect, the present application may provide a computer program or a computer program product, comprising instructions, which, when the program is executed by a computer, cause the computer to perform the above method.

[0210] In a fifth aspect, the present application may provide a data carrier signal carrying the above-mentioned computer program or computer program product.

[0211] The present application includes combinations of described aspects and preferred features unless such combinations are expressly impermissible or explicitly avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0212] Embodiments and experiments illustrating the principles of the present application will now be discussed with reference to the accompanying drawings, in which:

[0213] Figure 1 An object configured for storage and retrieval in an object-based data storage system is schematically illustrated.

[0214] Figure 2 A plurality of objects stored in an object-based data storage system including a plurality of buckets is schematically illustrated.

[0215] Figure 3 Schematically illustrates the separation of data and metadata in multiple objects into corresponding servers in an object-based data storage system.

[0216] Figure 4A and Figure 4B Metadata objects configured for storage and retrieval in an object-based data storage system are schematically illustrated.

[0217] Figure 5A and Figure 5B A plurality of metadata objects and associated objects stored in an object-based data storage system are schematically illustrated.

[0218] Figure 6 A plurality of merged metadata objects and associated objects stored in an object-based data storage system are schematically illustrated.

[0219] Figure 7 A merged metadata object and associated objects stored in an object-based data storage system are schematically illustrated.

[0220] Figure 8 A merged metadata object and associated objects and other individual objects stored in an object-based data storage system are schematically illustrated.

[0221] Fig. 9 A merged metadata object and associated objects stored in an object-based data storage system are schematically shown, along with several other individual metadata objects and associated objects, and along with several other individual objects.

[0222] Fig.10 A pair of related metadata objects is schematically shown, where one metadata object contains metadata identifying files and / or directories associated with stored data depending on an access control list (ACL), while the other metadata object contains the access control list as metadata.

[0223] Fig.11 The process of generating a hash by applying a cryptographic hash function to an information item containing metadata of one or more objects is schematically illustrated. DETAILED DESCRIPTION

[0224] Aspects and embodiments of the present application will now be discussed with reference to the accompanying drawings. Other aspects and embodiments will be apparent to those skilled in the art. All documents mentioned herein are incorporated herein by reference.

[0225] Figure 1 An object 1 configured for storage and retrieval in an object-based data storage system is schematically shown. The object includes a plurality of fields. The fields include a data field 3 configured to store data therein. The data may be in the form of a file, such as an image file, a video file, or a text file. The fields include a separate object ID attribute field 22 configured to store identification information associated with the object, such as an object name or a unique identifier (UID). Additional fields may include a separate attribute field 2 configured to store metadata generated by the object-based storage system, such as for recording attributes of the data stored in the data field 3.

[0226] Multiple objects (each with Figure 1 The object 1 shown in the form of can be stored in an object-based storage system, stored in a corresponding one of a plurality of "buckets" (4, 5, 6) within an overall storage space 7 (such as), such as in Figure 2 As is known in the art, a bucket is a logical container or compartment for storing objects. Users or systems (13, 14, 15) can create buckets within the storage space as needed. Buckets are usually associated with certain preset policies that determine what actions a user can perform on the bucket and on all objects in the bucket.

[0227] Existing object-based data storage systems typically explicitly (physically) separate metadata associated with each stored object from data (e.g., files, etc.) stored within the corresponding object. For example, such systems store metadata in a metadata server and store the file data separately in an object storage server. Figure 3 An example of this type of configuration is schematically shown, where the entire object storage space 8 is physically divided into an object storage server 9 and a separate metadata server 10. The data storage server only stores the data files 12 associated with the individual objects, while the metadata server 10 only stores the metadata 11 associated with each of the individual data files 12 stored in the data storage server 9. File system client software (14, 15) on the prior art system can interact with these different servers and abstract them to present a complete object-based file system to users and applications. The interface may include commands for creating and deleting objects, writing bytes to and reading bytes from individual objects, and setting properties on and getting properties from objects. However, this is highly resource intensive and can be inefficient.

[0228] In contrast to this configuration, in an embodiment of the present application, Figure 4A and Figure 4B As shown, the present application each provides a different method whereby metadata in addition to the file name of the object is stored in the file name attribute field of the object in the object-based storage system. This means that by implementing known operations of the object-based storage system (e.g., LIST operations), the stored metadata associated with the stored object can be retrieved directly from the stored object so as to retrieve the file name of the object when generating a list of stored objects. This is also particularly useful because it provides compatibility with file-based storage systems (e.g., POSIX-compliant systems) where equivalent directory "read" operations typically require that the metadata be provided.

[0229] In particular, refer to Figure 4A , schematically illustrates an object 160 configured for storage in an object-based data storage system. The object includes a data field 18 configured for storing data 21, and an attribute field area 17 including attribute fields configured for storing attributes of the object 160, the attributes of the object 160 including the attributes of the data 21 stored in the data field 18. The attribute field includes a plurality of attribute fields therein, including a file name field 19 configured for storing a file name of the object, and one or more other attribute fields 20, each of which is configured for storing other attributes 24 of the object. However, it is worth noting that the file name field 19 of the object 160 includes an information item 230, which includes metadata associated with the same object (e.g., associated with specific data 21 and / or associated with the object 160 as a whole). The information item 230 including the metadata performs the role of the file name of the object. In other words, the file name field may not include additional file name information, and may only include an information item including only the metadata 230. Even though the file name field 19 contains only the information item 230, the file name field 19 continues to be recognized as a file name field by the object-based data storage system in which the object 160 resides. Therefore, the metadata 230 stored within the file name field 19 will be retrieved directly from the object 160 by implementing known operations of the object-based storage system (e.g., LIST operations) to retrieve the file name of the object 160 when generating a list of stored objects.

[0230] In this manner, metadata contained within a filename attribute field of an object may include metadata associated with only that one object.

[0231] Figure 4B An object 161 according to another embodiment is schematically shown, which is configured to be stored in an object-based data storage system. The object 161 also includes a data field 18, which is configured to store data, but optionally, does not contain data (i.e., zero bytes, 211) within the data field. The object 161 also includes an attribute field area 17, which includes attribute fields configured to store attributes of the object 161. The attribute field includes a plurality of attribute fields therein, including a file name field 19 configured to store an information item 231 for performing a file name function of the object, and one or more other attribute fields 20 each configured to store other attributes 24 of the object. It is noteworthy that the file name field 19 of the object 161 includes an information item 231, which includes metadata of one or more other objects stored in the object-based data storage system, each object being not the object 161 itself. Therefore, the object 161 is used to store metadata that is not associated with itself or any data stored in its own data field (e.g., which may be zero bytes), but rather is associated with one or more other objects in the object-based data storage system. For this purpose, an object of this nature and function is referred to herein as a 'metadata object'. In this manner, metadata contained within the file name attribute field of the 'metadata object' may include metadata associated with at least one (preferably multiple) other objects. Once again, this information item 231 includes metadata within the metadata object and also performs the function of the file name of the object. In other words, the file name field may not contain additional file name information and may only contain information item 231 that includes only metadata. Even though the file name field 19 does not contain additional file name information, the file name field 19 continues to be recognized as a file name field by the object-based data storage system in which the object 161 is located. In this manner, the stored information item including metadata 231 within the file name field 19 related to other objects in the object-based storage system will be retrieved from the object 161 by implementing known operations of the object-based storage system (e.g., a LIST operation), which is used to retrieve the file name of the object 161 when generating a list of stored objects.

[0232] Metadata may include information associated with data stored in a data field of an object, the information including one or more of the following: file name; file path; file identification information. The file name attribute field may more generally be an object ID attribute field, which may include a file name attribute field or a unique identifier (UIDO) field, and the identification information associated with the corresponding object includes the file name or unique identifier (UIDO) associated with the corresponding object. Here, we refer to the file name attribute field and the file name, but it should be understood that the term includes references to the UIDO field and the UIDO.

[0233] Figure 5A An object-based data storage system for storing data in a plurality of objects is schematically shown. The data storage system includes a storage medium 80 configured to store a plurality of objects (250A, 250B, 250C, 250D, etc.), each object including a data field 18 configured to store data, and a separate object ID attribute field area 17 configured to store identification information associated with the corresponding object. These objects are described above with reference to Figure 4A As discussed, and includes a data field 18 containing data (e.g., files, etc.) and an attribute field 17 containing a file name field, the file name field containing the file name of the object represented in the form of an information item, the information item including metadata associated with the object. For example, a first object ('object 1'; 250A) stores an information item including metadata associated with the first object ('object 1'; 250A), used as the file name of the first object. Therefore, the information item including metadata is the file name associated with the first object and is stored in the file name attribute field of the first object. The data field of the first object contains at least some data (e.g., files, etc.) for storage in an object-based data storage system. Similarly, a second object ('object 2'; 250B) is dedicated to storing an information item including metadata associated with the second object ('object 2'; 250B), used as the file name of the second object. The information item including metadata is the file name associated with the second object and is stored in the file name attribute field area 17 of the second object. The data field of the second object contains at least some data (e.g., files, etc.) for storage in an object-based data storage system. Additional objects (250C, 250D, etc.) within the object-based data storage system (eg, each object) are similarly arranged to have associated information items including metadata contained within a filename attribute field of the object, serving as the filename of the object.

[0234] In this manner, information within the object ID attribute field of at least some (e.g., all) of the plurality of objects stored in the object storage medium 80 includes metadata within its file name attribute field, which is metadata associated with the object. The data processing device (13, 14, 15) includes a processor 13, which is configured to perform the processes and functions described above. A computer-readable medium (not shown) may include instructions stored thereon, which, when executed by the data processing device, cause the data processing device to perform the processes and functions described above. The data processing device is configured to access objects (250A, 250B, etc.) of the plurality of objects stored in the storage medium 80, at least by simply accessing the respective file name attribute fields of the objects to retrieve metadata information stored in the respective file name attribute fields of the objects. Suitably, such access to the objects may be performed by the processor 13 via a software application 14 and an application programming interface (API) 15. The processor 13 is configured to generate / create or overwrite a selected object (250A, 250B, ...) from among multiple objects stored in the storage medium 80, and to store an information item including metadata in a file name attribute field of the object (within the attribute field area 17), the metadata being used as a file name associated with the selected object (25A, 25B, ...).

[0235] Figure 5B Schematically illustrated is an alternative arrangement of an object-based data storage system for storing data in a plurality of objects. The data storage system includes a storage medium 80 configured to store a plurality of objects (16A, 16B, ..., etc.; 25A, 25B, ..., etc.), each of which includes a data field 18 configured to store data, and a separate object ID attribute field (within an attribute field area 17, including an attribute field) configured to store identification information associated with the respective object. However, as described above with reference to Figure 4B As discussed, there are multiple 'metadata objects' (16A, 16B, ..., etc.) among the multiple objects. These metadata objects include a data field 18 that optionally contains no data (i.e., zero bytes) and an attribute field 17 that includes a file name field, the file name field containing the file name of the file stored by another object and additional metadata associated with the other object and / or its stored files (i.e., in addition to the file name or ID). For example, a first metadata object ('metadata object 1'; 16A) is dedicated to storing a separate first other object ('object 1'; 25A) and / or its stored files together with the file name of the first other object and / or its stored files. The metadata and file name associated with the first other object and / or its stored files are stored in the file name attribute field of the first metadata object 16A. The attribute field of the first other object also contains the file name associated with the first other object, and the data field of the first other object contains at least some data (e.g., files, etc.) for storage within the object-based data storage system. Similarly, a second metadata object ('metadata object 2'; 16B) is dedicated to storing information associated with a separate second other object ('object 2'; 25B) and / or a file stored therein together with the file name of the second other object and / or the file stored therein. The metadata and file name associated with the second other object and / or the file stored therein are stored in the file name attribute field of the second metadata object 16B. The attribute field of the second other object also contains the file name associated with the second other object, and the data field of the second other object contains at least some data (e.g., files, etc.) for storage within the object-based data storage system. Additional objects (e.g., each object) within the object-based data storage system can be paired with an associated metadata object containing the metadata for that object.

[0236] In this way, information within the object ID attribute field of at least a metadata object among a plurality of objects stored in the object storage medium 80 includes metadata within its file name attribute field, which is different from identification information associated with the object. The data processing device (13, 14, 15) includes a processor 13, which is configured to perform the processes and functions described above. A computer readable medium (not shown) may include instructions stored thereon, which, when executed by the data processing device, cause the data processing device to perform the processes and functions described above. The data processing device is configured to access metadata objects (16A, 16B, etc.) among a plurality of objects stored in the storage medium 80, at least by accessing the corresponding file name attribute field of the metadata object to retrieve metadata information stored in the corresponding file name attribute field of the metadata object. Suitably, such access to the metadata object may be performed by the processor 13 via a software application 14 and an application programming interface (API) 15.

[0237] The processor 13 is configured to create / generate / generate or overwrite the selected metadata object (16A, 16B, ..., etc.) among the multiple objects stored in the storage medium 80. Alternatively or additionally, the processor 13 is selectively configured to generate a new metadata object (16A, 16B, ..., etc.) for storage among the multiple objects within the storage medium 80 (i.e., added to the multiple objects within the storage medium 80), and store the metadata and file name associated with another object (25A, 25B, ..., etc.) within the storage medium 80 within the file name attribute field (within the attribute field area 17) of the new metadata object. In this way, existing metadata objects can be overwritten and reused, or new metadata objects can be created as needed.

[0238] It should be noted that the information stored in the object file name attribute field of the metadata object may include metadata associated with more than one other object in the plurality of objects in the object-based data storage space 80. This means that the metadata object may store information items including metadata (also collectively performing the function of a file name) not only for one other object, as shown in FIG. 5 , but for multiple other objects, such as Figure 6 shown.

[0239] Figure 6 An example of such an optional arrangement of an object-based data storage system for storing data in a plurality of objects is schematically shown. The data storage system includes a storage medium 80 configured to store a plurality of objects (16C, 16D; 25A, 25B, ... 25n; 26A, 26B ... 26n), each of which includes a data field 18 configured to store data, and a separate object ID attribute field within an attribute field area 17 configured to store identification information associated with the respective object. However, among the plurality of objects there are two "metadata objects" (16C, 16D), as described above with reference to Figure 4B As discussed. Each of these metadata objects includes a data field 18 that optionally contains no data (i.e., zero bytes) and an attribute field 17 that includes a file name field containing a corresponding information item. The information item includes a corresponding plurality of other objects (25A, 25B...25n; 26A, 26B...26n) together with metadata associated with each of the plurality of other objects. In this manner, each corresponding information item is a composite information item that includes a plurality of component information items. Each component information item includes a file name (or file path) and associated metadata for a corresponding one of a plurality of objects stored in the storage medium 80 (e.g., metadata for files stored within the objects). For example, a first metadata object ('metadata object 1'; 16C) is dedicated to storing metadata associated with each of a first group of 'm' (m=integer) individual other objects ('object 1'; 25A: 'object 2'; 25B: ... 'object m'; 25m) and the file name of each corresponding other object. In object storage, because file names are immutable, an object-based data storage system can be configured to create new metadata objects to overwrite previous metadata objects, with the newer metadata objects taking precedence (e.g., also encoding timestamp / precedence information into the encoded metadata). This means that two metadata objects can reference the same object, with one taking precedence over the other. A single compressed encoding of metadata may not necessarily fit into the limited file name field (e.g., a 1024 character limit) of a single object, and so in the preferred embodiment discussed above, the object-based data storage system can be configured to split it into file names for multiple objects.

[0240] The metadata and file names associated with the first set of 'm' other objects are stored in the file name attribute field of the first metadata object 16C. The attribute field area 17 of the first other object 25A also contains the file name associated with the first other object, and the data field 18 of the first other object 25A contains at least some data (e.g., files, etc.) for storage within the object-based data storage system. The attribute field area 17 of the second other object 25B contains the file name associated with the second other object, and the data field 18 of the second other object 25B contains at least some data (e.g., files, etc.) for storage within the object-based data storage system. Similarly, each of the other objects up to and including the mth object 25m contains a corresponding attribute field area 17 and a data field 18 containing the corresponding file name and data for the object.

[0241] Similarly, the second metadata object ('Metadata Object 2'; 16D) is dedicated to storing a second group of 'n-m' (n=integer; n>m) separate other objects ('Object m+1'; 26A: 'Object m+2'; 26B: ... 'Object n'; 26n) and the file name of each corresponding other object. The metadata and file name associated with the first group of 'nm' other objects are stored in the file name attribute field of the second metadata object 16D. The attribute field area 17 of the first other object 26A of the second object group also contains the file name associated with the first other object, and the data field 18 of the first other object 26A of the second object group contains at least some data (e.g., files, etc.) for storage in the object-based data storage system. The attribute field area 17 of the second other object 26B in the second group contains the file name associated with the second other object, and the data field 18 of the second other object 26B contains at least some data (e.g., files, etc.) for storage in the object-based data storage system. Similarly, each of the other objects of the second group of objects, up to and including the (nm)th object 25n within the second group, contains a corresponding attribute field area 17 and a data field 18 containing the corresponding file name and data for that object.

[0242] In this manner, the corresponding information item in the object ID attribute field of each of only two metadata objects among the plurality of objects stored in the object storage medium 80 includes the metadata and the file name in its file name attribute field (e.g., Figure 4B Such as file name attribute field 19, as an example), which is associated with a group of other objects and is different from the identification information associated with the metadata object.

[0243] exist Figure 7 In another embodiment schematically shown, only one metadata object 16E among the multiple objects in the object storage medium 80 may contain a composite information item including metadata and a file name in its file name attribute field, the file name attribute field being associated with each of the other objects included in the first group "m" objects and the second group "nm" objects. In other words, the two groups of objects in question may be merged into one larger group served by one metadata item 16E.

[0244] exist Figure 8 In yet another embodiment schematically shown in , only one metadata object 16F among a plurality of objects within the object storage medium 80 may contain a composite information item including metadata and a file name within its file name attribute field associated with each of a subgroup "m" (m=integer) of other objects included only in the first group "m" of objects. The second group "nm" of objects has no group metadata object, and none of the objects within the second group of objects has an associated metadata object. In other words, the two groups of objects in question may be arranged in a mixed manner, with one group being served by only one metadata item 16F and the other group not being served by any metadata object.

[0245] It should be understood that other arrangements may be provided according to the present application, in which any one or more of the following permitted combinations of objects may be present simultaneously within the object storage medium 80:

[0246] (a) Figure 5A An object of the type shown (250A, 250B, etc.);

[0247] (b) Figure 5B Metadata objects of the indicated type (16A, 16B, ..., etc.);

[0248] (c) Figure 6 or Figure 7 or Figure 8 A group metadata object of the indicated type (16C, 16D, 16E, or 16F).

[0249] Fig. 9 An example of such an optional arrangement of an object-based data storage system for storing data in a plurality of objects is schematically shown. A storage medium 80 is configured to store a plurality of objects (16G, 16H; 16J; 25A, 25B, ... 25m; 26A, 26B; 264 ... 26n), each of which includes a data field 18 configured to store data, and a separate object ID attribute field (e.g., a data field 18) located within a larger attribute field area 17 configured to store identification information associated with the corresponding object. Figure 4B 19).

[0250] Among these multiple objects there are three 'metadata objects' (16G, 16H, 16J), as referenced above Figure 4B As discussed. These metadata objects include a data field 18 that optionally contains no data (i.e., zero bytes) and an attribute field 17 that includes a file name field that contains the file name of at least one other object. For example, the file name attribute field of the first metadata object of the three metadata objects 16G contains metadata and a file name associated with each of a plurality of other objects (25A, 25B, ..., 25M) that form a group of objects. In particular, the first metadata object ('metadata object 1'; 16G) is dedicated to storing a composite information item that includes metadata associated with each of 'm' (m=integer) individual other objects ('object 1'; 25A: 'object 2'; 25B: ... 'object m'; 25m) and the file name of each corresponding other object. The composite information item of metadata and file names associated with this group of 'm' other objects is stored in the file name attribute field (within the larger attribute field 17) of the first metadata object 16G. The attribute field area 17 of the first other object 25A in the group also contains a file name associated with the first other object, and the data field of the first other object 25A contains at least some data (e.g., a file, etc.) for storage in the object-based data storage system. The attribute field area 17 of the second other object 25B contains a file name associated with the second other object, and the data field of the second other object 25B contains at least some data (e.g., a file, etc.) for storage in the object-based data storage system. Similarly, each of the other objects up to and including the mth object 25m contains a corresponding attribute field area 17 and a data field 18 containing the corresponding file name and data for the object.

[0251] Similarly, a second metadata object ('Metadata Object 2'; 16H) is dedicated to storing metadata associated with a single other object ('Object m+1'; 26A) and a file name for that other object, and a third metadata object ('Metadata Object 3'; 16J) is dedicated to storing metadata associated with yet another single other object ('Object m+2'; 26B) and a file name for that other object. In addition, additional objects ('Object 4'; 264: ... 'Object n'; 26n) are stored within the object storage medium 80 without associated metadata objects. Each of these additional objects includes a file name attribute field containing a file name for only the corresponding object and a data field containing data for storage.

[0252] In this manner, information within the object ID attribute field (i.e., the file name attribute field) of each of only three metadata objects among the plurality of objects within object storage medium 80 includes a corresponding information item including metadata and a file name within its file name attribute field, the file name attribute field being associated with some but not all other objects within object storage medium 80, some of which are grouped (i.e., compound information items) and some of which are not grouped.

[0253] The data processing device (13, 14, 15) includes a processor 13, which is configured to perform the processes and functions as described above. A computer readable medium (not shown) may include instructions stored thereon, which when executed by the data processing device causes the data processing device to perform the processes and functions as described above. The data processing device is configured to access metadata objects (16C, 16D, 16E, 16F, 16G, 16H, 16J) from a plurality of objects stored in a storage medium 80, at least by simply accessing the information items stored in the individual file name attribute fields of the metadata objects to retrieve metadata information stored in the individual file name attribute fields of the metadata objects (within a larger attribute field area 17). Suitably, such access to information items stored in the metadata objects may be performed by the processor 13 via a software application 14 and an application programming interface (API) 15.

[0254] The processor 13 is configured to generate / create or overwrite a selected metadata object (16C, 16D, 16E, 16F, 16G, 16H, 16J) from among the plurality of objects stored in the storage medium 80, and store the metadata of the metadata object and a file name associated with another object (25A, 25B ... 25m; 26A, 26B ... 26n) in a file name attribute field (within the attribute field area 17). Alternatively or additionally, the processor 13 is selectively configured to generate a new metadata object (16C, 16D, 16E, 16F, 16G, 16H, 16J) for storing in the storage medium 80 (i.e., adding to) the plurality of objects stored in the storage medium 80, and store the new metadata object in a file name attribute field of the general attribute field area 17 of the new metadata object. Information items including metadata and an information item associated with another object (25A, 25B ... 25m; 26A, 26B ... 26n). In this way, existing metadata objects can be overwritten and rewritten, or new metadata objects can be created as needed.

[0255] Multiple objects including associated metadata objects can be arranged in a common storage bucket (not shown) within the storage medium 80, wherein the information stored in the file name attribute field of the metadata object includes metadata and / or identification information, and the metadata and / or identification information is associated with at least one other object among the multiple objects arranged in the common storage bucket or with multiple other objects.

[0256] As described above, the object data field of the metadata object is preferably an empty field that contains no data (i.e., zero bytes). However, in other arrangements, the object data field of the metadata object may contain a finite (non-zero) amount of data. The data may include additional metadata associated with the metadata object itself and / or associated with one or more other objects with which the metadata object is associated and / or associated with data contained in the data fields of one or more other objects with which the metadata object is associated. The data may include the full (wide) hash of the object referenced in the data portion for the metadata object. This facilitates situations where an application writes a new object directly to the object store, bypassing methods consistent with the present application. In this case, there is a risk of a hash collision between the new object name and a smaller (short) hash used in the encoded representation described herein.

[0257] refer to Fig.10 , the metadata stored within the file name attribute field of two metadata objects ('metadata object A'; 161 and 'metadata object B'; 162) may include information (231, 232) related to an access control list (ACL), which access control list contains a permission list associated with access to files stored within an object among multiple objects stored within the storage medium 80.

[0258] One of the two metadata objects includes a first metadata object ('metadata object A'; 161) containing metadata 231 (e.g., metadata items) within its file name attribute field 19 that identifies one or more files that are the subject of access control entries within an access control list. A second metadata object of the two metadata objects ('metadata object B') 162 contains the access control list 232 as metadata within its file name attribute field 19. One or more of the plurality of files following the access control list are contained in data fields of other objects stored in the storage medium 80. This arrangement allows the ACL to be updated solely and efficiently by accessing 'metadata object B' (162) without modifying 'metadata object A' (161).

[0259] The metadata 232 within the file name attribute field 19 of 'metadata object B' includes information defining an access control list (ACL) comprising an ordered list of a plurality of consecutive list entries. Each list entry contains access control information defining access controls applicable to data (e.g., files) stored within an object in the plurality of objects. In this example, the ACL for the files contained in the data fields of the plurality of separate objects comprises the following ordered list:

[0260] ACL List:

[0261] ACL entry#1

[0262] ACL entry #2…

[0263] ACL entry#n

[0264] The metadata item stored in the file name attribute field of 'metadata object A' relates to a data file to which one of the ACL entries relates. Entry: "ACL entry #1" contains access control information that defines access controls applicable to data (e.g., files) stored within one or more objects (e.g., inc. 'metadata object A') and relates to metadata items relating to files within the one or more objects. Similarly, entry: "ACL entry #2" contains access control information, etc. that defines access controls applicable to data (e.g., files) stored within one or more other objects and relates to metadata items relating to files within the one or more other objects.

[0265] A metadata item within an object such as 'metadata object A' includes a pre-stored hash of an access control list entry (e.g., 'hash[ACL entry#1]') within an access control list that is suitable for defining access control constraints to be applied to files referenced by the metadata item within 'metadata object A'. The pre-stored hash is generated by applying a pre-set hash function. The object-based storage system is configured to apply the same pre-set hash function (e.g., a cryptographic hash function) to access control entries within the access control list in 'metadata object B'. This generates a hash list:

[0266] Hash#1=Hash[ACL entry#1]

[0267] Hash#2=Hash[ACL entry#2]

[0268] …

[0269] Hash#n=Hash[ACL entry#n]

[0270] The object-based storage system is configured to identify which hash in the hash list derived from 'metadata object B' matches a pre-stored hash in 'metadata object A'. For example, 'metadata object A' may include a pre-stored hash = 'hash[ACL entry#1]', and this matches 'hash#1', thereby identifying that ACL entry #1 contains access control restrictions applicable to the file referenced by the metadata entry in 'metadata object A', which 'metadata object A' includes the pre-stored hash = 'hash[ACL entry#1]'.

[0271] Preferably, the metadata is stored in a compressed form. The identification information associated with each object may include a hash of one or more of the following: a file name, a file path, or file identification information associated with an object of the plurality of objects and containing the metadata.

[0272] The identification information associated with the respective object may include at least one hash of at least one metadata item of the plurality of metadata items associated with the respective one of the plurality of files to map the metadata item to the respective file name associated with the object of the plurality of objects.

[0273] Fig.11 The process of generating a hash for inclusion in an information item involved in the examples and embodiments described herein is schematically illustrated. The process includes obtaining a file name, file path, metadata item, or ACL entry (e.g., file path: images / (March-2022 / 0001.JPG), followed by a step 301 of applying a cryptographic hash function to the obtained file name, file path, metadata item, or ACL. This results in an output 302 of a hash value (e.g., 40-bit hash = 3402823669209384634633746074317682114551).

[0274] The metadata may include one or more symbolic links (also referred to as "Symlinks," or "SYLKs") that are configured to be interpreted and followed by the processor 13 as a path to a file or directory. For example, a symbolic link may include a "target_path" that defines a relative or absolute path to which the symbolic link points, and a "link_path" that defines the path of the symbolic link. Preferably, the one or more symbolic links are configured to conform to a POSIX-compliant operating system. The Portable Operating System Interface (POSIX) is a series of standards for maintaining compatibility between operating systems, which is well known in the art.

[0275] In this manner, as embodied in the above examples, in a preferred embodiment, the object-based data storage system implements a method (e.g., by the processor 13), the method comprising the following steps:

[0276] Step 1: Provide an object-based data storage medium that is configured to store (and optionally, has stored) multiple objects, each object including a data field in which data is stored, and a separate object ID attribute field (e.g., a file name attribute field) that stores identification information associated with the object.

[0277] Step 2: Generate one or more objects including in its object ID attribute field (e.g., file name attribute field), the object containing an information item used as an ID (e.g., file name) of the generated object, and containing metadata different from the object ID information associated with the generated object. The metadata includes one or more of the following:

[0278] (a) with the generated object in question (e.g. Figure 4A ; Figure 5A ). Optionally, all objects stored in the data storage medium are objects generated in this manner;

[0279] (b) metadata (e.g., metadata item) associated with a single other object among a plurality of objects stored in an object-based data storage medium (e.g., Figure 4B , Figure 5B ). Optionally, only some (but not all) of the objects to be stored in the data storage medium are objects (e.g., 'metadata objects') generated in this manner, wherein each generated object serves an existing object in the data storage medium;

[0280] (c) with data stored in an object-based data storage medium (e.g. Figures 6 to 9 ). Optionally, only one or only some (but not all) of the objects to be stored in the data storage medium are objects (e.g., 'metadata objects') generated in this way, wherein each generated object serves an existing object in the data storage medium.

[0281] Step 3: Store one or more generated objects in a storage medium.

[0282] Step 4: Access at least one generated object from a plurality of objects stored in a storage medium to retrieve at least information (e.g., information items) stored in its object ID attribute field (e.g., file name attribute field), thereby retrieving metadata (e.g., metadata items) stored therein.

[0283] In this manner, the method may generate new objects for storage in an object-based data storage medium.

[0284] The method may comprise the further steps of:

[0285] Step 5: Accessing one or more selected objects stored in the storage medium.

[0286] Step 6: Storing (e.g., overwriting) metadata in an object ID attribute field of each of the one or more accessed objects, the object ID attribute field being different from the object ID information associated with the accessed object. The metadata includes one or more of the following:

[0287] (a) related to the object of discussion (e.g. Figure 4A ; Figure 5A );

[0288] (b) metadata (e.g., metadata item) associated with a single other object among a plurality of objects stored in an object-based data storage medium (e.g., Figure 4B , Figure 5B );

[0289] (c) with data stored in an object-based data storage medium (e.g. Figures 6 to 9 ) metadata (e.g., metadata items) associated with multiple other objects in a plurality of objects.

[0290] In this way, the method can overwrite existing objects within the object-based data storage medium. This can be a modification Figure 1 An object of the type shown, making it Figure 4A or Figure 4B An object of the indicated type, or modify Figure 4A or Figure 4B Existing objects of the type shown have modified metadata (230, 231) within their corresponding object ID attribute fields (eg, file name attribute fields).

[0291] In "step 2" or "step 6", the metadata may include information associated with the data stored in the data field 21 of the object. The data may include one or more files, and the associated information contained in the metadata may include: a file name; a file path for the file; file identification information for the file; a timestamp (e.g., creation time, modification time, or access time); a user ID ('UID'); a group ID ('GID'), access permissions (e.g., access control information); one or more file attribute bits. A file attribute is a piece of information associated with a file or directory that includes additional data about the file itself or its contents. For example, a byte may store the attributes of a file. Each specific attribute may be assigned to a specific bit of the byte. To enable a certain attribute, the system may assign a bit value of (e.g.) 1 ('1') to the corresponding bit, indicating the 'on' state of the attribute. The attribute bit may correspond to one or more of the following attributes: executable; symbolic link; directory bit; setuid bit; setgid bit.

[0292] The features disclosed in the preceding description, or in the following claims, or in the accompanying drawings, in their specific form, or as means for performing the disclosed functions, or as methods or processes for obtaining the disclosed results, may, if appropriate, be used alone or in any combination of these features to implement the present application in its different forms.

[0293] Although the present application has been described in conjunction with the above exemplary embodiments, many equivalent modifications and variations are apparent to those skilled in the art when this disclosure is given. Therefore, the above exemplary embodiments of the present application are considered to be illustrative rather than restrictive. Various changes may be made to the described embodiments without departing from the spirit and scope of the present application.

[0294] For the avoidance of any doubt, any theoretical explanations provided herein are intended to enhance the reader's understanding. The applicant does not wish to be bound by any of these theoretical explanations.

[0295] Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

[0296] Throughout this specification, including the following claims, unless the context requires otherwise, the words "comprise" and "comprising" and variations such as "comprises", "including" and "containing", will be understood to imply the inclusion of stated integers, or steps, or groups of integers or steps but not the exclusion of any other integers, or steps, or groups of integers or steps.

[0297] It must be noted that, as used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Herein, ranges may be expressed as starting from "about" one particular value, and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when a value is expressed as an approximation by using the antecedent "about", it will be understood that the particular value forms another embodiment. The term "about" in relation to a value is optional and refers to, for example, + / - 10%.

[0298] Example

[0299] Example 1

[0300] As mentioned above, the following algorithm provides an illustrative example of a detailed method for generating metadata items. It should be understood that the present application is not limited to the steps of these algorithms.

[0301] I - Create a merged payload

[0302] A-Calculation Steps

[0303] 1- Calculate a unique GID list from a GID list. The GIDs may be encoded with (eg, containing) Unicode symbols such as and LTF8 Unicode symbols.

[0304] Calculate 'ngid': the number of bytes required to store the index

[0305] 2- Calculate a unique UID list from the UID list. The UID may be encoded with (eg, contain) Unicode symbols, ie, LTF8 Unicode symbols.

[0306] Calculate 'nuid': the number of bytes required to store the index

[0307] 3- Sort the list of ctime (i.e., creation time) and its corresponding index

[0308] 4- Compute the dual of the order from the existing order by increasing ctime using the index from step (IA-3)

[0309] 5- Calculate adjacent differences in the ctime list in (IA-4) sort. The calculated differences may be encoded with (e.g., including) Unicode symbols such as and LTF8 Unicode symbols.

[0310] 6- Compute the list: for each mtime (ie, modification time), the difference to the ctime (with sign) of the same entry is reordered by (IA-4).

[0311] 7- Compute the list: for each GID, it corresponds to the index in the list (IA-1) in the 'ngid' bytes reordered by the (IA-4) bijection

[0312] 8- Compute the list: for each UID it corresponds to the index in the 'nuid' bytes of the list (IA-2) reordered by the (IA-4) bijection

[0313] 9- Compute the list: For each hash, compute the minimum number of bits that makes this hash unique compared to the hash of every object record in the directory that is older than the merge (even if not part of the merge) by (IA-4) bijective reordering.

[0314] B-Encoding Step

[0315] 1 - The number of entries to write into the merged payload. This number may be encoded using (e.g., containing) Unicode symbols, such as LTF8 Unicode symbols.

[0316] 2- Append the combined payload to the combined timestamp. UNIX time is rounded to the lowest 5 minutes and stored in 3 bytes.

[0317] 3- Added to the merged payload list for IA-9, packaged as follows:

[0318] -For each hash:

[0319] - 7 bits for hash size

[0320] - X bits of the hash itself (X being the value above)

[0321] - At the end, zero fill to the end of the current byte

[0322] 4- The pre-compressed payload number written to the unique GID. This number may be encoded with (e.g., contain) Unicode symbols, such as LTF8 Unicode symbols.

[0323] 5-Pre-compressed payload number appended to the unique UID. This number may be encoded with (e.g., contain) Unicode symbols, such as LTF8 Unicode symbols.

[0324] 6 - Append to the pre-compressed payload list a 1-byte mask of entries reordered via (IA-4) bijection.

[0325] 7 - Append to the pre-compressed payload list of entries 2-byte file mode reordered via (IA-4) bijection.

[0326] 8 - List of pre-compressed payloads appended to (IA-7).

[0327] 9-Append to the pre-compressed payload list of (IA-8).

[0328] 10 - List of pre-compressed payloads appended to (IA-9).

[0329] 11- List of pre-compressed payloads appended to (IA-5).

[0330] 12 - A list of pre-compressed payloads appended to (IA-6) in the following format:

[0331] - Byte for the sign: 0 if ctimes is greater than M time, 1 otherwise

[0332] - Absolute difference. This number may be encoded using (eg, containing) a Unicode symbol, such as an LTF8 Unicode symbol.

[0333] 13 - Append to the pre-compressed payload list of 4-byte file ACL hashes reordered via (IA-4) bijection.

[0334] 14 - Append to the pre-compressed payload list of 4-byte directory default ACL hashes reordered via (IA-4) bijection.

[0335] 15 - Append to the pre-compressed payload list of 4-byte position ID hashes reordered via (IA-4) bijection.

[0336] 16-Append to the pre-compressed payload list from (IA-1).

[0337] 17-Append to the pre-compressed payload list from (IA-2).

[0338] 19 - Compress the pre-compressed payload using compression (e.g., ZStd compression) and append the result to the merged payload.

[0339] II - Split the combined payload into combined filenames

[0340] A-Computing Head

[0341] 1 - Write 1 byte version to the header

[0342] 2- Append 1 byte padding to the header

[0343] 3- Calculate the hash of the combined payload (e.g., Blake3 hash) and append the first 3 bytes to the header

[0344] B- Calculate the merged file name

[0345] Do the following until the end of the merged payload is reached. Start with part ID 0.

[0346] 1-Write the (II-A) header to the pre-encoded file name

[0347] 2- Append the section ID to the pre-encoded file name. Increment section ID

[0348] 3- Append the maximum amount that will not exceed the character limit after encoding to the pre-encoded file name.

[0349] 4- Encode the byte stream from the pre-encoded filename into a character stream using 91 / 128 bit mapping, adding / separators every 256 characters if required by the cloud provider. Append the result to the combined filename

[0350] III - Creating a partially merged payload

[0351] A-Calculation Steps

[0352] All steps 10 from IA - Sort each attribute timestamp in increasing order.

[0353] 11- Take the minimum value in the list (III-A-10).

[0354] 12- Calculate list: For each attribute, the timestamp is calculated to the minimum of the difference from step (I11-A-11). This number can be encoded using (e.g., containing) Unicode symbols, such as LTF8 Unicode symbols.

[0355] B-Encoding Step

[0356] 1 - The number of entries written to the pre-encoded payload. This number may be encoded using (e.g., containing) Unicode symbols, such as LTF8 Unicode symbols.

[0357] 2- Append the combined minimum timestamp to the pre-encoded payload. This number may be encoded with (e.g., contain) Unicode symbols, such as LTF8 Unicode symbols.

[0358] 3 - Pre-coded payload list appended to IA-9, bits are packed as follows:

[0359] -For each hash:

[0360] - 7 bits for hash size

[0361] - X bits of the hash itself (X being the value above)

[0362] - At the end, zero fill to the end of the current byte

[0363] 4-Write the pre-compressed payload list of (II-A-12).

[0364] 5 The pre-compressed payload number written to the unique GID. This number may be encoded using (e.g., containing) Unicode symbols, such as LTF8 Unicode symbols.

[0365] 6-Pre-compressed payload number appended to the unique UID. This number may be encoded with (e.g., contain) Unicode symbols, such as LTF8 Unicode symbols.

[0366] 7 - Append to the pre-compressed payload list a 1-byte mask of entries reordered via (IA-4) bijection.

[0367] 8 - Append to the pre-compressed payload list of entries 2-byte file mode reordered via (IA-4) bijection.

[0368] 9 - List of pre-compressed payloads appended to (IA-7).

[0369] 10 - List of pre-compressed payloads appended to (IA-8).

[0370] 11- List of pre-compressed payloads appended to (IA-9).

[0371] 12- List of pre-compressed payloads appended to (IA-5).

[0372] 13 - A list of pre-compressed payloads appended to (IA-6) in the following format:

[0373] - Byte for the sign: 0 if ctimes is greater than M time, 1 otherwise

[0374] - Absolute difference. This number may be encoded using (eg, containing) a Unicode symbol, such as an LTF8 Unicode symbol.

[0375] 14 - Pre-compressed payload list appended to 4-byte file ACL hash reordered via (IA-4) bijection.

[0376] 15 - Append to the pre-compressed payload list of 4-byte directory default ACL hashes reordered via (IA-4) bijection.

[0377] 16 - Append to the pre-compressed payload list of 4-byte position ID hashes reordered via (IA-4) bijective.

[0378] 17-Append to the pre-compressed payload list from (IA-1).

[0379] 18 - Append to pre-compressed payload list from (IA-2).

[0380] 19- Compress the pre-compressed payload using compression (e.g., ZStd compression) and append the result to the pre-encoded payload.

[0381] 20- Convert the byte stream from the pre-encoded payload to the merged payload using 91 / 128 bit mapping.

[0382] IV - Split the combined payload into combined filenames:

[0383] Same steps as (II) using a partially merged payload instead of a merged payload.

[0384] Payload format

[0385] Not merged :

[0386] [version] 1 byte

[0387] [padding] 1 byte

[0388] [payload_hash]

[0389] [part_ID]

[0390] [timestamp if first part]

[0391] [payload] (divided into multiple parts) [...]

[0393] An example of a single unmerged information item for / Images / March-2022 / 0001.JPG could be:

[0394] / Images / March-2022 / .meta / [Full hash of / Images / March-2022 / 001.JPG][part number(1 / 1)][timestamp][payload]

[0395] The portion of the information item " / Images / March-2022" is an example of what is known in the art as a "prefix" of a file path. The "prefix" portion of a file path corresponds to the portion of the file path up to, but not including, the file name of the file to which the file path relates. The file name will be found at the end of the file path. In this sense, the "prefix" of a file path can be thought of as a truncation of the file path where the file name has been removed or does not exist. In this example, the full file path is " / Images / March-2022 / 001.JPG" and this is the file path for the file "001.JPG", so the "prefix" of the file path for his file is " / Images / March-2022".

[0396] The " / .Meta" portion of the information item is an optional portion of the information item which may optionally be combined or replaced with selected non-mapped Unicode symbols which may be included to help identify the source or origin of the information item if desired. This may be appended to the file path prefix if desired, as shown in the example.

[0397] Appended to the file path prefix (or to the Unicode symbol if present) is a hash (e.g., a cryptographic hash) of the full file path. In this simple example, the appended hash is the hash of the file path " / Images / March-2022 / 001.JPG".

[0398] Appended to the hash of the file path is a partial identifier.

[0399] The payload can be, for example:

[0400] [bitmask][metadata1][metadata2]

[0401] The payload may be compressed. [bitmask] may be a bit mask that corresponds to or identifies which type of information is conveyed by the metadata contained in the payload. As an example, the bit mask may be an ordered sequence of n bits (e.g., n=5), where the position of the bit within the sequence identifies the type of metadata (information type), and the value of the bit identifies whether metadata of that type is present within the payload (e.g., within the metadata appended to the bit mask). The ordering of different types of metadata within the payload corresponds to the ordering of the bits within the bitmap. As a simple illustration, a bit mask value of [bitmask=01100] indicates that the two pieces of metadata "[metadata1][metadata2]" appended to the bit mask include possible five pieces of metadata. This is indicated by the presence of two bit values ​​"1" and three bit values ​​"0". The position of the first bit value "1" indicates that the first metadata corresponds to a group ID ("GID"). The position of the second bit value "1" indicates that the first metadata corresponds to a modification time ("mtime") value:

[0402] [metadata1][metadata2]=[gid][mtime]

[0403] Thus, as an illustrative example, if an object-based storage system performs a LIST operation to list the content stored corresponding to the file path prefix " / Images / March-2022", the result is as follows:

[0404] / Images / March-2022 / .meta / ajkshkajshdkla[1][payload part1]

[0405] / Images / March-2022 / .meta / fkjsdfkjhasfsv[2][payload part2]

[0406] / Images / March-2022 / .meta / ajkshkajshdkla[3][payload part3]

[0407] / Images / March-2022 / .meta / ajkshkajshdkla[2][payload part2]

[0408] / Images / March-2022 / .meta / fkjsdfkjhasfsv[1][payload part1]

[0409] The object-based storage system is configured to identify a common hash "fkjsdfkjhasfsv" between two of the five listed contents (e.g., the hash of / Images / March-2022 / 001.JPG):

[0410] / Images / March-2022 / .meta / fkjsdfkjhasfsv[1][payload part1]

[0411] / Images / March-2022 / .meta / fkjsdfkjhasfsv[2][payload part2]

[0412] The identified part numbers [payload part 1] and [payload part 2] identify the first of the two listed items as a first part of a larger payload, and the second of the two listed items as a second part of a larger payload. The object-based storage system is configured to reassemble the two payload parts into a combined payload: [payload] = [payload part 1] & [payload part 2].

[0413] The object-based storage system is also configured to identify a common hash "ajkshkajshdkla" in three of the five listed contents (e.g., the hash of / Images / March-2022 / 002.JPG):

[0414] / Images / March-2022 / .meta / ajkshkajshdkla[1][payload part1]

[0415] / Images / March-2022 / .meta / ajkshkajshdkla[2][payload part2]

[0416] / Images / March-2022 / .meta / ajkshkajshdkla[3][payload part3]

[0417] The identified part numbers [payload part 1], [payload part 2], and [payload part 3] identify the three listed items as the first, second, and third parts of a larger payload. The object-based storage system is configured to reassemble the three payload parts into a combined payload: [payload] = [payload part 1] & [payload part 2] & [payload part 3].

[0418] In other examples, entries in the output of a LIST operation may contain hashes that are not common to any other hashes in the list, and thus may correspond to payloads that have not been split into parts.

[0419] merge:

[0420] [version] 1 byte

[0421] [padding] 1 byte

[0422] [payload_hash]

[0423] [part_ID]

[0424] [payload] (divided into multiple parts)

[0425] [table size]

[0426] [timestamp]

[0427] [short (eg, truncated) hashes]

[0428] / / The following is compressed

[0429] [how_many_GIDs]

[0430] [how_many_UIDs]

[0431] [masks]

[0432] [modes]

[0433] [GID indexes]

[0434] [UID indexes]

[0435] [ctimes]

[0436] [ctime to mtime diff]

[0437] [ACL hash(file)]

[0438] [ACL hash (default)]

[0439] [location]

[0440] [unique GIDs]

[0441] [unique UIDs]

[0442] As a simple but illustrative example, an example of a merge of information items for three file paths: / Images / March-2022 / 0001.JPG, / Images / March-2022 / 0002.JPG, and / Images / March-2022 / 003.JPG may be as follows:

[0443] / Images / March-2022 / .meta / [Hash of payload][part number(1 / 3)][part ofpayload split over parts]

[0444] / Images / March-2022 / .meta / [Hash of payload][part number(2 / 3)][part ofpayload split over parts]

[0445] / Images / March-2022 / .meta / [Hash of payload][part number(3 / 3)][part ofpayload split over parts]

[0446] Here, the payload has a size such that it is divided into three information items, which together contain the combined payload of the information items divided into three parts. In another example, the payload may have a size such that it does not need to be split over multiple information items in this way. In this case, there will be only one part number (e.g., "[part number (1 / 1)]").

[0447] Notably, the encoding of the merged information item differs in that it has been appended to the file path prefix (or to the optional Unicode symbol / .Meta / , if present), and the hash of the entire payload is split across multiple information items (e.g., "[hash of payload]"), as opposed to the hash of the file path (e.g., " / Images / March-2022 / 001.JPG]"), which is used in the one unmerged information item discussed above. In particular, "[Hash of payload]" does not correspond to the hash of any one "[part of payload split over parts]" contained within the information item in question, but rather, "Hash of payload]" corresponds to the hash of the entire payload of which each "[part of payload split over parts]" forms a part. In other words, each "[part of payload split over parts]" may be combined together into a larger original (unsplit) payload, and the "[hash of payload]" corresponds to the hash of the larger original (unsplit) payload. The object-based storage system may be configured to split the larger original payload into its parts, and combine the split payload parts upon subsequent retrieval.

[0448] This hashing of the larger original (unsplit) payload allows the object-based storage system to identify multiple information items that share the same hash as being associated with the same split payload (e.g., the three information items shown above would have the same "[hash of payload]" value).

[0449] This hash of the payload is, in turn, appended by a part number that identifies a specified one of multiple ordered parts of the payload in question (e.g., "[part number (1 / 3)]", "[part number (2 / 3)]", "[part number (3 / 3)]"). The part number is then appended by the payload. An object-based storage system can be configured to read and interpret the part numbers and identify the payload appended thereto as a specified part within a sequential set of a specified number of parts that can be combined together into a larger payload. An object-based storage system can be configured to combine parts of a separate payload according to the order indicated by the part numbers.

[0450] The object-based storage system can be configured to read and interpret hashes of payloads appearing within a consolidated information item (e.g., "[hash of payload]") as a means of identifying other consolidated information items of the object-based storage system that contain different portions of the payload that are to be reassembled into a reconstructed payload when retrieved. The object-based storage system can be configured to read and interpret payload part numbers accordingly (e.g., "[part number(1 / 3)]") to indicate the order of the components of the payload and the sequence with which those payload components should be reassembled when reconstructing the entire payload.

[0451] Thus, as an illustrative example, if an object-based storage system performs a LIST operation to list the content stored corresponding to the file path prefix " / Images / March-2022", the result is as follows:

[0452] / Images / March-2022 / .meta / abkjhktjshdkla[1 / 3][payload part1]

[0453] / Images / March-2022 / .meta / fkjrajljhasfsv[1 / 2][payload part1]

[0454] / Images / March-2022 / .meta / abkjhktjshdkla[2 / 3][payload part2]

[0455] / Images / March-2022 / .meta / abkjhktjshdkla[3 / 3][payload part3]

[0456] ……wait.

[0457] Here, "[Hash of payload]" is "abkjhktjshdkla", which identifies that those listed entries that share this hash have a partial payload that corresponds to one larger payload split over three parts. "[Hash of payload]" identified as "fkjrajjhasfsv" does not correspond to this larger payload, but rather to another larger payload.

[0458] As mentioned above, the [payload]([payload]) can include different metadata and corresponding bitmasks. For example, the payload can include:

[0459] [bitmask][metadata1][metadata2][metadata3]...etc.

[0460] The merged information item contains a composite information item containing information derived from the multiple component information items involved in the merge process. Therefore, the payload may also include a hash of the file paths associated with each component information item merged therein. This may be in the form of a list. A simple example is:

[0461] [Short hash of / Images / March-2022 / 001.JPG][Short hash of / Images / March-2022 / 002.JPG][Short hash of / Images / March-2022 / 003.JPG]

[0462] Partial Merge:

[0463] [version] 1 byte

[0464] [padding] 1 byte

[0465] [payload_hash]

[0466] [part_ID]

[0467] [payload] (split into multiple parts)

[0468] [table size]

[0469] [timestamp] (ie, this is different than the structure used for merging above)

[0470] [minimum record timestamp]

[0471] [short(truncated)hashes][not unique]

[0472] / / The following is compressed

[0473] [record timestamps deltas to the minimum]

[0474] [how_many_GIDs]

[0475] [how_many_UIDs]

[0476] [masks]

[0477] [modes]

[0478] [GID indexes]

[0479] [UID indexes]

[0480] [ctimes]

[0481] [ctime to mtime diff]

[0482] [ACL hash(file)]

[0483] [ACL hash(default)]

[0484] [location]

[0485] [unique GIDs]

[0486] [unique UIDs]

[0487] Single property entry:

[0488] / / OUTPUT FORMAT PRIOR TO SPLITTING:

[0489] / / mask:1

[0490] / / OPTIONAL: / / access_mode:3

[0491] / / UID:4

[0492] / / GID:4

[0493] / / mtime:8

[0494] / / ctime:8

[0495] / / ACL:4(PART_HASH_SIZE)

[0496] / / default_ACL:4(PART_HASH_SIZE)

[0497] / / location ID:2

[0498] / / AFTER SPLITTING:

[0499] / / multi_part_hash:4(PART_HASH_SIZE)

[0500] / / part_ID:2

[0501] / / part 1only-creation timestamp:4

[0502] / / payload:up to available space

[0503] / / FINAL FILENAME TO BE APPENDED TO DESTINATION ATTRIBUTE SUBDIR: / / <encoded object hash> / <encoded: <x><part idx=0><part payload>>

[0504] / / <encoded object hash> / <encoded: <k><part idx=1><part payload>>

[0505] / / ...

[0506] POSIX ACL encoding:

[0507] / / OUTPUT FORMAT PRIOR TO SPLITTING:

[0508] / / payload_size:4

[0509] / / payload

[0510] / / AFTER SPLITTING:

[0511] / / multi_part_hash:4(PART_HASH_SIZE).Hash is not caclulated withpayload size included,just the payload

[0512] / / part_ID:2

[0513] / / part 1only-creation timestamp:4

[0514] / / payload:up to available space

[0515] / / FINAL FILENAME TO BE APPENDED TO DESTINATION ATTRIBUTE SUBDIR:

[0516] / / Al<encoded:<multipart_hash><part idx=0><part payload>>

[0517] / / Al<encoded:<multipart_hash><part idx=1><part payload>>

[0518] / / ...< / k> < / x>

Claims

1. A computer-implemented object-based data storage system for storing data in a plurality of objects, the data storage system comprising: a storage medium configured to store the plurality of objects; Wherein, each of the plurality of objects comprises a plurality of fields, the plurality of fields comprising: a data field configured for storing the data therein; and a separate object ID attribute field configured for storing identification information associated with the object; wherein the information stored in the object ID attribute field of at least one object of the plurality of objects includes metadata other than the identification information associated with the at least one object; and A processor is configured to access the at least one object from the plurality of objects stored in the storage medium to retrieve at least information stored in an object ID attribute field thereof, thereby retrieving the metadata.

2. An object-based data storage system according to any one of the preceding claims, wherein: The processor is configured to: access a selected object among the multiple objects stored in the storage medium to store the metadata in its object ID attribute field, and / or generate an object containing the metadata in its object ID attribute field for storage in the multiple objects stored in the storage medium.

3. An object-based data storage system according to any one of the preceding claims, wherein: The information stored in the object ID attribute field of the at least one object includes metadata associated with at least one other object from the plurality of objects other than the at least one object.

4. An object-based data storage system according to any one of the preceding claims, wherein: The information stored in the object ID attribute field of the at least one object includes identification information associated with at least one other object from among the plurality of objects other than the at least one object.

5. An object-based data storage system according to any one of the preceding claims, wherein: The metadata includes: information associated with the data stored in the data field of an object in the plurality of objects, including one or more of: a file name, a file path, and file identification information.

6. An object-based data storage system according to any one of the preceding claims, wherein: The identification information associated with the respective object includes a hash of one or more of a file name, a file path, or file identification information associated with the object of the plurality of objects and containing the metadata.

7. The object-based data storage system according to claim 6, wherein: The identification information associated with the respective object includes at least one hash of at least one metadata item of a plurality of metadata items associated with a respective one of a plurality of files to map the metadata item to a respective file name associated with an object of the plurality of objects.

8. An object-based data storage system according to any one of the preceding claims, wherein: The metadata comprises an access control list (ACL) containing a permission list associated with access to files stored within an object of the plurality of objects, and wherein the plurality of objects comprises at least one other object containing the files to which the access control list relates.

9. A computer-implemented object-based data storage method for storing data in a plurality of objects, the method comprising: Providing a plurality of objects, wherein each of the plurality of objects comprises a plurality of fields, the plurality of fields comprising: a data field configured to store data therein; and a separate object ID attribute field configured to store identification information associated with the object; wherein the information stored in the object ID attribute field of at least one object of the plurality of objects includes metadata other than the identification information associated with the at least one object; storing the plurality of objects on a storage medium; The at least one object from the plurality of objects is accessed, by the processor, to retrieve at least information stored within an object ID attribute field thereof, thereby retrieving the metadata.

10. The method according to claim 9, comprising: Through the processor, a selected object among the multiple objects stored in the storage medium is accessed to store the metadata in its object ID attribute field, and / or an object containing the metadata is generated in its object ID attribute field for storage in the multiple objects stored in the storage medium.

11. The method according to any one of claims 9 to 10, comprising: Within the information stored within the object ID attribute field of the at least one object, metadata associated with at least one other object from among the plurality of objects other than the at least one object is stored.

12. The method according to any one of claims 9 to 11, comprising: In the information stored in the object ID attribute field of the at least one object, identification information associated with at least one other object from among the plurality of objects other than the at least one object is stored.

13. The method according to any one of claims 9 to 12, wherein: The metadata includes: information associated with data stored in the data field of an object in the plurality of objects, information including one or more of: a file name, a file path, and file identification information.

14. The method according to any one of claims 9 to 13, wherein: The identification information associated with the corresponding object includes: a hash of one or more of a file name, a file path, or file identification information associated with an object among the multiple objects and containing the metadata, and / or includes: a hash of a file path associated with an object among the multiple objects, and / or includes: at least one hash of at least one metadata item among multiple metadata items associated with a corresponding one of the multiple files to map the metadata item to a corresponding file name associated with the object among the multiple objects.

15. The method according to any one of claims 9 to 14, wherein: The metadata comprises an access control list (ACL) containing a permission list associated with access to files stored within an object of the plurality of objects, and wherein the plurality of objects comprises at least one other object containing the files to which the access control list relates.

16. A data processing device comprising a processor configured to execute the method according to any one of claims 9 to 15.

17. A computer readable medium comprising instructions stored thereon which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 9 to 16.

18. A computer program or a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 9 to 16.

19. A data carrier signal carrying the computer program or computer program product of claim 18.

Images