Methods, devices, systems and service platforms for querying and statistically analyzing object storage information

Abstract

This application discloses a method, apparatus, system, and service platform for querying and statistically analyzing object storage information. The application establishes an interface on the object storage server that supports specific query types for object storage data. Based on this interface, the object storage server can obtain task requests submitted by users carrying file constraint information of the required query type. It can then directly query the metadata of files that meet the requirements from the metadata set corresponding to the target storage bucket and perform statistical analysis on the query results to meet the user's data storage statistical needs. This eliminates the need for users to perform statistical analysis on the indiscriminately returned file information of the entire storage bucket through coding, reducing the coding requirements on users and improving execution efficiency. Furthermore, since this application returns statistical results, compared to traditional technologies that require an interface to return file information for the entire storage bucket, it significantly reduces the operation time of the interface.

Description

Technical Field

[0001] This application belongs to the field of object storage technology, and in particular relates to a method, apparatus, system and service platform for querying and statistically analyzing object storage information. Background Technology

[0002] Object storage is a distributed storage solution for the internet, supporting file reading, writing, and management anytime, anywhere via HTTP (Hypertext Transfer Protocol) or HTTPS (Hypertext Transfer Protocol over Secure Socket Layer). It supports standard REST APIs (Application Programming Interfaces) and utilizes buckets as containers to provide clients with unlimited storage space through a flat storage architecture. Object storage is a highly reliable, highly available, low-cost, and infinitely scalable storage method, suitable for storing massive amounts of unstructured data, and its resources can be elastically expanded.

[0003] Users (such as enterprise users) often have a need to collect statistics on the data storage status of object storage. In traditional technologies, users need to use the interface provided by object storage to query and obtain the file information of the entire bucket, such as detailed metadata information such as the file name, size, and creation time of the files stored in the entire bucket. Based on this, the data storage status of object storage can be obtained by encoding the file information obtained by using the interface.

[0004] The applicant found that traditional technologies have drawbacks such as high interface operation time, low execution efficiency of data storage statistics, and the requirement for users to have strong coding skills. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, system and service platform for querying and statistically analyzing object storage information, which overcomes at least some of the shortcomings of traditional solutions and provides convenience for users to query and statistically analyze the data storage status based on object storage.

[0006] To achieve the above objectives, the following technical details are disclosed:

[0007] A method for querying and statistically analyzing object storage information includes:

[0008] The query request submitted by the client is obtained through the query interface pre-set on the object storage server; the request parameters of the task request include at least file constraint information corresponding to the target query type and bucket information of the target storage bucket.

[0009] Query the file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket to obtain the metadata of at least one target file;

[0010] Perform statistical processing on the metadata of the at least one target file to obtain statistical results;

[0011] Generate response information including the statistical results, and send the response information back to the client through the query interface.

[0012] Optionally, in the above method, the target query type is a file size query type or a file prefix query type;

[0013] If the target query type is a file size query type, the file constraint information includes one or more values ​​of the file size;

[0014] If the target query type is a file prefix query type, the file constraint information includes one or more values ​​of the file prefix.

[0015] Optionally, in the above method, the statistical processing of the metadata of the at least one target file includes:

[0016] The total number of files corresponding to each target file in the metadata obtained from the statistical query;

[0017] And / or, the total file size of each target file corresponding to the metadata obtained from the statistical query.

[0018] Optionally, in the above method, the request parameters further include at least one of query type information, response result type information, and signature authentication information obtained by signing the various information included in the request parameters; wherein:

[0019] The query type information is used to indicate the target query type;

[0020] The response result type information is used to indicate one of the following response result types:

[0021] It also returns the total number of files and the total size of the target file;

[0022] Returns the total number of files in the target file;

[0023] Returns the total size of the target file.

[0024] Optionally, in the above method, before querying file metadata that conforms to the file constraint information from the metadata set corresponding to the target storage bucket, the method further includes:

[0025] The signature authentication information is verified to verify the user identity of the client.

[0026] If the verification is successful, the step of querying file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket is triggered.

[0027] If the verification fails, the task request for the query task is rejected, and the client is provided with corresponding indication information indicating that the signature verification failed.

[0028] Optionally, in the above method, after querying the file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket to obtain the metadata of at least one target file, the method further includes:

[0029] Cache the metadata of at least one of the target files;

[0030] Upon receiving a request to export metadata, a metadata file including the metadata of the at least one target file is generated based on the cached metadata of the at least one target file, and the metadata file is transmitted to the client so that the client can export the metadata file;

[0031] And / or, upon detecting that the client submits the task request for the query task again, the task request is responded to based on the cached metadata of the at least one target file.

[0032] Optionally, in the above method, the object storage server is deployed on a cloud platform, and a console is also built on the cloud platform as the entry point of the query interface; wherein, the console is used to: provide a visual graphical interface to client users, and to call the query interface according to the task request of the obtained query task, and to generate a metadata file based on the cached corresponding metadata and transmit it to the client according to the obtained export request for metadata;

[0033] The process of generating a metadata file including the metadata of the at least one target file based on the cached metadata of the at least one target file includes:

[0034] The console loads the cached metadata of the at least one target file and generates a metadata file including the metadata of the at least one target file according to a predetermined file format.

[0035] The response processing for the task request based on the metadata of the at least one target file cached includes:

[0036] The console queries the metadata of the at least one target file in the cache, and processes the task request in response based on the queried metadata of the at least one target file.

[0037] Optionally, in the above method, the console is further used to: provide the data owner with the function of authorizing the data owner's interface operation permissions for the query interface to the target user through the visual graphical interface;

[0038] The target user is either a sub-user of the data owner or another user distinct from the data owner.

[0039] A query and statistics device for object storage information, comprising:

[0040] The acquisition module is used to acquire the task request of the query task submitted by the client through the query interface pre-set on the object storage service side; the request parameters of the task request include at least the file constraint information corresponding to the target query type and the bucket information of the target storage bucket;

[0041] The query module is used to query file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket, and obtain the metadata of at least one target file;

[0042] The statistics module is used to perform statistical processing on the metadata of the at least one target file to obtain statistical results;

[0043] The response processing module is used to generate response information including the statistical results and to send the response information back to the client through the query interface.

[0044] A service platform includes: a server-side object storage; the server-side object storage is equipped with a query interface;

[0045] The object storage server is used for:

[0046] The task request submitted by the client is obtained through the query interface pre-set on the object storage service side; the request parameters of the task request include at least file constraint information corresponding to the target query type and bucket information of the target storage bucket;

[0047] Query the file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket to obtain the metadata of at least one target file;

[0048] Perform statistical processing on the metadata of the at least one target file to obtain statistical results;

[0049] Generate response information including the statistical results, and send the response information back to the client through the query interface.

[0050] Optionally, the aforementioned service platform also includes a console as the entry point for the query interface;

[0051] The console is used to: provide a visual graphical interface to client users, and to invoke the query interface based on the task request of the obtained query task.

[0052] Optionally, in the above service platform, the object storage server is further used for:

[0053] Cache the metadata of at least one of the target files;

[0054] The console is also used for:

[0055] Upon receiving a request to export metadata, a metadata file including the metadata of the at least one target file is generated based on the cached metadata of the at least one target file, and the metadata file is transmitted to the client so that the client can export the metadata file;

[0056] And / or, upon detecting that the client submits the task request for the query task again, the task request is responded to based on the cached metadata of the at least one target file.

[0057] Optionally, in the above service platform, the console is also used for:

[0058] The visual graphical interface provides the data owner with the function of authorizing the target user to access the query interface.

[0059] The target user is either a sub-user of the data owner or another user distinct from the data owner.

[0060] A query and statistics system for object storage information, comprising:

[0061] At least one client;

[0062] The service platform as described in any of the preceding items;

[0063] The service platform provides the client with query and statistics services for object storage information, at least through the server that has the query interface.

[0064] As can be seen from the above solution, the object storage information query and statistics method, apparatus, system, and service platform provided in this application set up a unified query interface on the object storage server that supports specific query types for object storage data. Based on this interface, the object storage server can obtain task requests submitted by users carrying file constraint information of the required query type, and can directly query the file metadata that meets the requirements in the metadata set corresponding to the target storage bucket, and perform statistics on the query results to meet the user's data storage statistics needs. Users are not required to perform statistical processing on the file information of the entire storage bucket returned indiscriminately through coding, reducing the coding ability requirements for users and improving execution efficiency. Furthermore, since this application returns statistical results, compared with the traditional method of returning the file information of the entire storage bucket through an interface, the operation time of the interface can be greatly reduced. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating the object storage information query and statistics method provided in this application;

[0067] Figure 2(a) is an example of the interface configuration under the file prefix query type provided in this application;

[0068] Figure 2(b) is an example of the interface configuration under the file size query type provided in this application;

[0069] Figure 3 This is an example diagram of the client's response information interface provided in this application;

[0070] Figure 4 This is another flowchart illustrating the object storage information query and statistics method provided in this application;

[0071] Figure 5 This is another flowchart illustrating the object storage information query and statistics method provided in this application;

[0072] Figure 6 This is an example of a list of metadata exported via the console provided in this application;

[0073] Figure 7 This is a structural diagram of the object storage information query and statistics device provided in this application;

[0074] Figure 8 This is a structural diagram of the service platform provided in this application;

[0075] Figure 9 This is a structural diagram of the object storage information query and statistics system provided in this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] The current object storage provides a file information query interface. However, the applicant found that this interface only supports users to obtain file information for the entire bucket in the object storage, such as detailed metadata information like filename, size, and creation time of the files stored in the entire bucket. Moreover, the interface returns the metadata information of the files in the bucket in the form of files, and each returned file corresponds one-to-one with the data file in the bucket. Each returned file contains the metadata information of the data file in the bucket that corresponds to that file. Due to the flow control limit of the query interface, such as returning a maximum of 1,000 files at a time, the interface operation is very costly. For example, if the number of files in the bucket space is in the hundreds of millions, then this operation will be very time-consuming.

[0078] Users (e.g., enterprise users) often have statistical needs regarding the data storage status of object storage-based systems. For example, in data migration scenarios, users need to know the total number / total size of all files in a bucket that meet a fixed size or a fixed prefix, to facilitate data migration deployment. Traditionally, to address this need, users must not only obtain the file information of the entire bucket using the interface, but also further encode and statistically analyze this information to understand the data storage status in object storage. Therefore, traditional technologies require users to have strong coding skills and reduce execution efficiency.

[0079] To overcome at least some of the shortcomings of traditional solutions, embodiments of this application disclose a method, apparatus, system, and service platform for querying and statistically analyzing object storage information.

[0080] The object storage information query and statistics method disclosed in this application can be applied to cloud platforms, such as public or private cloud platforms. Optionally, an object storage architecture and related object storage services on the local side of the architecture (e.g., services for querying and statistics of metadata of storage buckets) are deployed on the cloud platform. In this embodiment, the related object storage services on the local side of the object storage are referred to as the object storage server.

[0081] See Figure 1 The flowchart illustrates a query and statistics method for object storage information according to an embodiment of this application. The method includes at least:

[0082] Step 101: Obtain the task request for the query task submitted by the client through the query interface pre-set on the object storage server; the request parameters of the task request shall include at least the file constraint information corresponding to the target query type and the bucket information of the target storage bucket.

[0083] The target bucket is the bucket where the file for which file information is to be queried is located.

[0084] The bucket information of the target bucket may include, but is not limited to, the bucket name and / or number, which are used to uniquely identify the bucket.

[0085] To facilitate user queries and statistics of data from object storage, this application provides a unified and universal query interface on the object storage server. This interface supports specific query types for object storage data. This interface can be, but is not limited to, a RESTful API (Application Programming Interface). Optionally, it can provide users with interactive functionality between the query interface in the cloud via a command-line interface or a graphical interface on the client device.

[0086] Preferably, the query interface is a synchronous interface.

[0087] For the aforementioned query interface deployed on the object storage server, this application embodiment further designs the interface as follows:

[0088] 11) Request parameters and request syntax;

[0089] 12) Response parameters and response content.

[0090] The request parameters include at least the bucket information of the target storage bucket (e.g., the bucket name) and file constraint information corresponding to the target query type. In addition, in some embodiments, the request parameters may also include any one or more of the following: query type, response result type, and signature authentication information.

[0091] For example, in this embodiment, the query types include the following two:

[0092] 21) File prefix;

[0093] 22) File size.

[0094] In the file prefix query type, the configured file constraint information includes the value of the file prefix, which is used to indicate the metadata of all files in a bucket that match the configured prefix. In the file size query type, the configured file constraint information includes the value of the file size, which is used to indicate the metadata of all files in a bucket that match the configured file size.

[0095] The configured file size can be a fixed value or a range, and supports setting one or more fixed values ​​or ranges. During subsequent queries, if the query target (the metadata of files in the bucket) matches one of these values ​​or ranges, it is considered to meet the criteria and will be selected as the query target. The configured file prefix is ​​a fixed prefix, such as "123", and supports configuring one or more fixed prefixes. Optionally, different file size values / ranges or different fixed prefixes can be separated by delimiters such as commas, spaces, vertical bars, and forward slashes.

[0096] It is easy to understand that the query types are not limited to the two mentioned above. For example, they can also include query types such as file keywords. That is, by configuring one or more keywords, you can indicate the metadata of all files in a certain storage bucket that match the configured keywords. For this implementation, you can add "keyword" dimension information to the metadata database of the object storage file to support file information queries for keyword query types.

[0097] For example, the response result types include the following three:

[0098] 31) Simultaneously return the total number of target files and the total file size that meet the file constraint information;

[0099] 32) Returns the total number of target files that meet the file constraint information;

[0100] 33) Returns the total size of the target file that meets the file constraint information.

[0101] The following table provides detailed descriptions of at least some of the request parameters, including bucket information, query type, file constraint information, response result type, and signature authentication information, as shown in Table 1 below.

[0102] Table 1

[0103]

[0104] Furthermore, this embodiment also provides an example of the request syntax for request parameters:

[0105] https: / / BucketName.${domain name of the region where the bucket is located} / object

[0106] type= <prefix>

[0107] &prefixvalue = <123,234>

[0108] &sizevalue = <1024-1024>

[0109] &resstyle= <0>

[0110] Authorization:{SignatureValue}

[0111] Wherein, BucketName represents the name of the target storage bucket, and the meanings of type, prefixvalue, sizevalue, resstyle, and Authorization can be found in the explanation in Table 1.

[0112] Based on the above design of the request parameters and request syntax of the query interface in the embodiments of this application, when a client user has a query and statistics requirement for a specific file in a certain storage bucket (e.g., a file that meets a certain prefix value or a file that meets a fixed size value), the client interface of the query interface can configure the required target query type, file constraint information corresponding to the target query type, and return result type, etc. Preferably, this embodiment provides a visual graphical interface on the client, where the user can select the required storage bucket (storage space) according to the requirements (e.g., click the storage bucket icon or mark) and further configure other parameter information such as query type. See Figures 2(a) and 2(b) for details, which respectively show interface configuration examples under the file prefix and file size query types.

[0113] Then, the configured parameters are submitted to the query interface via the network in the form of a task request.

[0114] The object storage server responds by using this query interface to obtain the query task request submitted by the client and parse the various task parameters it carries.

[0115] Step 102: Query the metadata of files that match the above file constraint information from the metadata set corresponding to the target storage bucket to obtain the metadata of at least one target file.

[0116] In object storage, in addition to using buckets to store data files, a metadata database is also built to store metadata information of the files in the buckets.

[0117] Metadata information in the metadata database includes, but is not limited to, the filename, size, creation time, and file prefix of the corresponding file. The file prefix reflects the directory in which the file is stored within the bucket. Different buckets correspond to different namespace metadata sets in the metadata database. The metadata set for a given namespace of a bucket can be, but is not limited to, one or more metadata tables in the metadata database.

[0118] On the object storage server, the query task request is obtained through the query interface. After parsing the various request parameters it carries, the corresponding namespace metadata set is determined from the metadata database based on the parsed bucket information, such as the bucket name of the target bucket. This metadata set includes the metadata corresponding to all files in the target bucket.

[0119] Next, based on the parsed query type and file constraint information, the metadata of each file that satisfies the file constraint information is retrieved from the metadata set corresponding to the target bucket, thus obtaining the metadata of at least one target file. For example, in the file size query type, the metadata of all files that meet the required file size value or range is retrieved from the metadata set; in the file prefix query type, the metadata of all files that meet the required prefix value is retrieved from the metadata set, and so on.

[0120] Step 103: Perform statistical processing on the metadata of at least one of the above target files to obtain statistical results.

[0121] Subsequently, statistical processing is performed directly on the metadata of at least one of the aforementioned target files obtained from the query on the object storage server. This statistical processing includes, but is not limited to, any one or more of the following:

[0122] a. Calculate the total number of files corresponding to each target file obtained from the statistical query metadata;

[0123] Specifically, the total number of metadata entries obtained from the query (each metadata entry corresponds to a data file in a data bucket) can be counted, and the total number of metadata entries can be used as the total number of files in each target file that meet the query requirements.

[0124] b. Calculate the total file size of each target file corresponding to the metadata obtained from the statistical query.

[0125] Specifically, the file size of each metadata record obtained from the query can be accumulated, and the accumulated result is the total file size of each target file.

[0126] When the request parameters of the task request include the response return type, the specific processing type for statistical processing can be determined based on the response return type. For example, if the response return type is to return both the total number of files and the total file size, both the total number of target files that meet the file constraint information and the total file size will be counted simultaneously; if the response return type is to return the total number of files, only the total number of target files that meet the file constraint information will be counted; and if the response return type is to return the total file size, only the total file size of target files that meet the file constraint information will be counted.

[0127] Step 104: Generate response information including statistical results and send the generated response information back to the client through the query interface.

[0128] After performing statistical processing on the metadata obtained from the query and obtaining the statistical results, the results are further encapsulated into response information in the format required by the query interface, and then the response information is fed back to the client through the query interface.

[0129] In contrast to the request parameters mentioned above, this embodiment also designs response parameters for the query interface of this application. The response parameters include at least statistical result information, such as the total number of target files and / or the total file size. In addition, optionally, any one or more of the following information may be included, but not limited to: the request identifier of this task request, the return result container, and the operation identifier of this task request (used to indicate the query type such as file size or file prefix).

[0130] For detailed descriptions of each parameter, please refer to Table 2.

[0131] Table 2

[0132] Parameter name illustrate RequestId Request Data ID, a unique request identifier sumresult Returns the result container, i.e., the parent node. Type The requested operation identifier represents the type of operation. Count Total number of files that meet this condition Storage Total file size that meets this condition, in bytes

[0133] The response message may, but is not limited to, using a JSON (JavaScript Object Notation) structure. Below is an example of a JSON-structured response message:

[0134]

[0135] For the above response information, please refer to Figure 3 Furthermore, an example of the client's response information interface is provided.

[0136] Furthermore, after the interface call is successful and a valid response information including the above statistical results is provided to the user, a status code indicating a successful interface call, such as HTTP status code (Status) 200, can be returned to the client. Conversely, if the interface call fails due to certain factors, a status code indicating a failure is returned, such as a 4xx or 5xx HTTP status code (Status), to inform the user of the interface call status.

[0137] Factors that may cause API call failures include, but are not limited to: incorrect parameter value configuration in the request parameters, incorrect parameter value type configuration, reaching the flow control limit of the query API, internal system errors (such as system crashes, database deletion, API request timeouts, etc.) or internal service exceptions (such as service timeouts), etc.

[0138] See Table 3 below for detailed descriptions of various types of interface errors.

[0139]

[0140]

[0141]

[0142] As can be seen from the above scheme, the object storage information query and statistics method provided in this application embodiment sets up a unified query interface on the object storage server that supports specific query types for object storage data. Based on this interface, the object storage server can obtain the task request submitted by the user carrying file constraint information of the required query type, and can directly query the file metadata that meets the requirements in the metadata set corresponding to the target storage bucket, and perform statistics on the query results to meet the user's data storage statistics needs. This eliminates the need for the user to perform statistical processing on the file information of the entire storage bucket returned indiscriminately through coding, reducing the coding ability requirements for the user and improving execution efficiency. Furthermore, since this application returns statistical result information, compared with the traditional method of returning the file information of the entire storage bucket through an interface, it can greatly reduce the operation time of the interface.

[0143] In some embodiments, optionally, when the client configures request parameters based on the query interface to submit a query task to the query interface, in addition to configuring parameters such as query type, file constraint information, and response result type, it further performs signature processing on these parameters, and carries the signature authentication information obtained after signing along with the query type and other parameters in the task request and submits it to the query interface.

[0144] The signature processing employs a standard asymmetric key-based signature procedure. This involves using a predefined hash algorithm to hash all configured parameters, such as query type, file constraint information, and response result type. The hash result is then encrypted using the client user's private key to obtain the signature authentication information, which is used to verify the client user's identity. Optionally, the client user's private key is the private key from the public-private key pair assigned to the user when creating their account. The private key is stored on the client side, while the public key is synchronized and stored on the object storage server for authentication when the client user submits a query task through the query interface.

[0145] Therefore, as Figure 4 As shown, the object storage information query and statistics method disclosed in this application embodiment may further include the following processing between step 101 and step 102:

[0146] Step 401: Verify the signature authentication information to verify the client's user identity; if the verification is successful, proceed to step 102; if the verification fails, proceed to step 402.

[0147] After the object storage server obtains the query task request through the query interface and parses out the signature authentication information, it can use the standard verification process based on asymmetric key signature to verify whether the client's user identity is an recognized legitimate identity.

[0148] Specifically, in reverse of the signing process, the server first calls the pre-stored public key corresponding to the client that initiated the request, and uses this public key to decrypt the signature authentication information to obtain the decryption result. At the same time, the agreed hash algorithm consistent with that used during signing is used to perform hash operations on various parameters in the task request, such as query type, file constraint information, and response result type. The hash operation result is then compared with the decryption result. If they match, the verification is successful, indicating that the user's identity is recognized as legitimate. Otherwise, if they do not match, the verification fails.

[0149] If the verification is successful, proceed to step 102 to continue the subsequent query and statistical processing flow.

[0150] Step 402: Reject the task request for the current query task and send a corresponding indication message to the client indicating that the signature verification failed.

[0151] If the verification fails, the task request is rejected, and the reason for the failure is identified, such as signature mismatch, invalid key, missing key, or no valid signature. Based on the reason for the failure, the client is given the corresponding instruction information, such as "Signature mismatch, please check the signature algorithm", "Invalid key", "Request rejected because no key", or "Request rejected because no valid signature" as shown in Table 3.

[0152] This embodiment verifies the identity of the client user by signing and verifying the request parameters in the task request, which can effectively avoid non-compliant task requests, such as preventing illegal task requests from users without proper permissions.

[0153] In some embodiments, optionally, see Figure 5 The object storage information query and statistics method disclosed in this application embodiment may further include the following processing after step 102:

[0154] Step 501: Cache the metadata of at least one of the above target files obtained from the query.

[0155] Optionally, the metadata of each target file that meets the task requirements can be cached on the object storage server. Specifically, the mapping information between the task information of this query (such as query type, file constraint information, etc.) and the metadata of each target file that meets the task requirements can be cached.

[0156] Step 502: When a request to export metadata is received, a metadata file including the metadata of the at least one target file is generated based on the cached metadata of the at least one target file, and the metadata file is transmitted to the client so that the client can export the metadata file.

[0157] This application embodiment also constructs a console on the cloud platform. This console is located in front of the object storage server and is used as the entry point for the query interface set by the object storage server.

[0158] This console is used at least to provide a visual graphical interface to client users and to invoke the query interface based on the task request of the obtained query task.

[0159] In addition, the console can generate metadata files based on the cached metadata according to the client's request to export metadata, and then transfer the metadata files to the client.

[0160] Specifically, in practical applications, in addition to the need for information statistics on all target files in a storage bucket that meet the requirements of a fixed file size / fixed prefix, users may also have the need to obtain the metadata of all target files that meet these requirements. To address this need, this embodiment specifically sets up an export function for the metadata list that meets the task requirements in the console, and sets up an export control corresponding to this function on the client's visual graphical interface, such as an export button / tool ​​item. Users can use this control to send a request to the console to export the metadata list corresponding to this query task.

[0161] After receiving the request, the console loads the metadata of each target file cached for this query task and generates a metadata file that includes at least all the loaded metadata according to a predefined file format, such as generating an Excel metadata list containing information for each metadata entry. Optionally, the generated metadata file may also include the query type corresponding to this query task, file constraint information (e.g., the range of file size values), return result type, total number of files / total file size, etc. See the reference for details. Figure 6 The example provided.

[0162] Step 503: When it is detected that the client submits the above query task request again, the task request is responded to based on the metadata of the cached at least one of the above target files.

[0163] As the entry point for the query interface, the console receives the task request information submitted by the client before the query interface. To improve the response rate of the task request, in this embodiment, when the console receives the task request information, it does not directly submit the task request information to the query interface and then hand it over to the object storage server for processing. Instead, it first performs a metadata information hit in the cache of the object storage server based on the task information (e.g., query type, file constraint information). If the cache contains metadata information that meets the task requirements, the console directly provides the client with corresponding information statistics and metadata export services based on the hit metadata information. That is, it bypasses the object storage server's query and statistics of object storage information such as metadata database, thereby improving the task response rate.

[0164] Conversely, if no match is found, the console will further submit the task request information to the query interface, which will then hand the task request over to the object storage server for processing.

[0165] Specifically, when performing metadata hit processing in the cache, the received task information can be compared with the cached task information, and the metadata corresponding to the matching cached task information can be selected as the hit objects.

[0166] In this way, for a query task submitted by the client, if it is the first time the task is submitted, the storage service server will provide the necessary query and statistics services for the task through the query interface. If it is not the first time the task is submitted, the query interface and the storage service server can be bypassed, and the console can directly handle the task based on the cache hit, thereby simplifying the processing flow and improving the response speed.

[0167] In practical applications, the "task information-metadata" correspondence information in the cache can also be released according to the predetermined cache release strategy, such as releasing metadata information whose request frequency is lower than the frequency lower limit or whose cache duration exceeds the duration upper limit, so as to balance the performance of cache resource occupation and response rate as much as possible.

[0168] This application embodiment associates cached task information with file metadata information that meets the task requirements, and sets up a console on the cloud platform as the entry point for the query interface. This further provides the client with a channel to obtain detailed file information (i.e., file metadata) that meets the requirements. Compared with the prior art, which returns detailed information of data files to the client in a one-to-one correspondence format through the query interface, this export channel based on the console (without needing to go through the query interface) allows users to quickly export detailed information of files that meet the task requirements. It provides different fast acquisition channels for two types of file information: summary (e.g., total file size / total number of files) and detailed (e.g., metadata information list), which facilitates user operation.

[0169] In addition, this embodiment further improves the response rate of query tasks by combining the console's buffer hit processing for task response.

[0170] In some embodiments, optionally, the console described above can also be used to: provide the data owner with the function of authorizing the target user to access the query interface through a visual graphical interface provided by the client.

[0171] The target users here can refer to the data owner's sub-users or other users who are different from the data owner.

[0172] The data owner's access permissions to the query interface include, but are not limited to, the data owner's permissions to add, delete, modify, and query the data they own in the cloud.

[0173] In this embodiment, the console further provides a function to authorize the data owner's interface operation permissions, and on the client's visual graphical interface, operation controls matching this function are set, such as authorization buttons, toolboxes, or dialog boxes. The data owner can operate these controls and perform necessary information configurations, such as configuring the target user to be authorized, the data to be authorized, and the interface operation permissions for the data to be authorized, such as query / modify, or query / modify / delete, to authorize the corresponding interface operation permissions of the configured data to the target user, such as authorizing a sub-user of the data owner (such as a sub-account under the company's main account) or other users different from the data owner (such as an account of another company that has a cooperative relationship with the company).

[0174] This embodiment provides the data owner with the function of authorizing interface operation permissions through the console, which further improves the convenience of client users to operate the query interface, thereby facilitating the interface operation for client users.

[0175] Corresponding to the above-described method for querying and statistically analyzing object storage information, this application also discloses a device for querying and statistically analyzing object storage information, such as... Figure 7 As shown, the device includes:

[0176] The acquisition module 701 is used to acquire the task request of the query task submitted by the client through the query interface pre-set on the object storage service side; the request parameters of the task request include at least the file constraint information corresponding to the target query type and the bucket information of the target storage bucket;

[0177] The query module 702 is used to query the metadata of files that meet the file constraint information from the metadata set corresponding to the target storage bucket, and obtain the metadata of at least one target file;

[0178] The statistics module 703 is used to perform statistical processing on the metadata of at least one of the above target files to obtain statistical results;

[0179] The response processing module 704 is used to generate response information including the statistical results and to send the response information back to the client through a query interface.

[0180] In one embodiment, the target query type is either a file size query type or a file prefix query type;

[0181] If the target query type is a file size query type, the above file constraint information includes one or more values ​​of the file size;

[0182] If the target query type is a file prefix query type, the above file constraint information includes one or more values ​​of the file prefix.

[0183] In one embodiment, the statistics module 703 is specifically used for:

[0184] The total number of files corresponding to each target file in the metadata obtained from the statistical query;

[0185] And / or, the total file size of each target file corresponding to the metadata obtained from the statistical query.

[0186] In one embodiment, the request parameters further include at least one of query type information, response result type information, and signature authentication information obtained by signing the various information included in the request parameter information; wherein:

[0187] The query type information is used to indicate the target query type mentioned above;

[0188] The response result type information is used to indicate one of the following response result types:

[0189] It also returns the total number of files and the total size of the target file;

[0190] Returns the total number of files in the target file;

[0191] Returns the total size of the target file.

[0192] In one embodiment, the above-described apparatus further includes a signature verification module, used for:

[0193] The signature authentication information in the task request is verified to verify the user's identity on the client side.

[0194] If the verification passes, the query module 702 is triggered to perform the process of querying the metadata of files that match the file constraint information from the metadata set corresponding to the target storage bucket;

[0195] If the verification fails, the task request to query the task is rejected, and the client is sent back with the corresponding indication that the signature verification failed.

[0196] In one embodiment, the above-described apparatus further includes:

[0197] The caching module is used to cache the metadata of at least one target file that meets the file constraint information obtained from the query;

[0198] An export processing module and / or an accelerated response module are provided, wherein the export processing module is used to generate a metadata file including the metadata of the at least one target file based on the cached metadata of the at least one target file when an export request for metadata is received, and transmit the metadata file to the client so that the client can export the metadata file; the accelerated response module is used to respond to the task request based on the cached metadata of the at least one target file when a task request for the query task is detected to be submitted by the client again.

[0199] In one embodiment, the object storage service client is deployed on a cloud platform, which also has a console as the entry point for the query interface. The console is used to: provide a visual graphical interface to the client user, and to call the query interface according to the task request of the obtained query task, and to generate a metadata file based on the cached corresponding metadata and transmit it to the client according to the obtained export request for metadata.

[0200] The export processing module described above, when generating a metadata file including the metadata of the at least one target file based on the cached metadata of the at least one target file, is specifically used to: load the cached metadata of the at least one target file into the console, and generate a metadata file including the metadata of the at least one target file according to a predetermined file format.

[0201] The aforementioned accelerated response module, when responding to a task request based on the metadata of the cached at least one target file, is specifically used to: query the metadata of the cached at least one target file through the console, and respond to the task request based on the queried metadata of the at least one target file.

[0202] In one embodiment, the console is also used to: provide the data owner with the function of authorizing the data owner's interface operation permissions for the query interface to the target user through a visual graphical interface;

[0203] The target users are: the data owner's sub-users or other users who are different from the data owner.

[0204] The object storage information query and statistics device disclosed in this application is relatively simple to describe because it corresponds to the object storage information query and statistics methods disclosed in the above method embodiments. For related similarities, please refer to the description of the corresponding method embodiments above, and it will not be described in detail here.

[0205] This application also discloses a service platform that can be deployed in the cloud, such as on a public cloud or a private cloud. See [link to relevant documentation]. Figure 8 The service platform includes at least an object storage server 801, wherein the object storage server is equipped with a query interface 802;

[0206] The aforementioned object storage server 801 is specifically used for:

[0207] The task request submitted by the client is obtained through the query interface 802 pre-configured on the object storage service side; the request parameters of the task request include at least the file constraint information corresponding to the target query type and the bucket information of the target storage bucket.

[0208] Query the metadata of files that match the file constraint information from the metadata set corresponding to the target bucket to obtain the metadata of at least one target file;

[0209] Perform statistical processing on the metadata of at least one of the aforementioned target files to obtain statistical results;

[0210] Generate a response message including the above statistical results, and send the response message back to the client through query interface 802.

[0211] In one embodiment, such as Figure 8 As shown, the aforementioned service platform also includes a console 803, which serves as the entry point for the query interface 802;

[0212] Console 803 is used to: provide a visual graphical interface to client users, and to invoke query interface 802 based on the task request of the obtained query task.

[0213] In one embodiment, the server 801 of the object storage is further configured to: cache the metadata of the at least one target file obtained from the query;

[0214] The aforementioned console 803 is also used for:

[0215] Upon receiving a request to export metadata, based on the cached metadata of at least one of the aforementioned target files, a metadata file including the metadata of the at least one target file is generated, and the metadata file is transmitted to the client so that the client can export the metadata file;

[0216] And / or, upon detecting that the client submits the task request for the above query task again, the task request is responded to based on the cached metadata of at least one of the above target files.

[0217] In one embodiment, the console 803 is further used for:

[0218] Through a visual graphical interface, the system provides data owners with the ability to authorize target users with the data owner's access rights to query interface 802.

[0219] The target users are: the data owner's sub-users or other users who are different from the data owner.

[0220] The service platform disclosed in this embodiment also corresponds to the object storage information query and statistics method disclosed in the previous method embodiments. The functions of each component of the service platform, as well as their respective processing flow or mutual collaborative interaction, can be found in the relevant descriptions of the above method embodiments, and will not be repeated here.

[0221] In addition, this application also discloses a query and statistics system for object storage information, see [link to relevant documentation]. Figure 9 The provided system architecture specifically includes:

[0222] At least one client 901;

[0223] Service platform 902 as disclosed in the corresponding embodiments above;

[0224] The service platform 902 further includes an object storage server and a console, wherein the object storage server is deployed with a query interface, and the console is used as the entry point for the query interface.

[0225] The service platform can provide the client with the function of querying and statistically analyzing object storage information by executing the object storage information query and statistics method provided in any of the embodiments of the above methods.

[0226] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0227] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0228] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0229] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0230] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / prefix>

Claims

1. A method for querying and statistically analyzing object storage information, characterized in that, include: The task request for the query task submitted by the client is obtained through the query interface that is pre-set on the object storage server. The request parameters of the task request include at least file constraint information corresponding to the target query type and bucket information of the target storage bucket; Query the file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket to obtain the metadata of at least one target file; Perform statistical processing on the metadata of the at least one target file to obtain statistical results; Generate response information including the statistical results, and send the response information back to the client through the query interface.

2. The method according to claim 1, characterized in that, The target query type is either a file size query type or a file prefix query type; If the target query type is a file size query type, the file constraint information includes one or more values ​​of the file size; If the target query type is a file prefix query type, the file constraint information includes one or more values ​​of the file prefix.

3. The method according to claim 1, characterized in that, The statistical processing of the metadata of the at least one target file includes: The total number of files corresponding to each target file in the metadata obtained from the statistical query; And / or, the total file size of each target file corresponding to the metadata obtained from the statistical query.

4. The method according to claim 1, characterized in that, The request parameters also include at least one of query type information, response result type information, and signature authentication information obtained by signing the information included in the request parameters; wherein: The query type information is used to indicate the target query type; The response result type information is used to indicate one of the following response result types: It also returns the total number of files and the total size of the target file; Returns the total number of files in the target file; Returns the total size of the target file.

5. The method according to claim 4, characterized in that, Before querying file metadata that matches the file constraint information from the metadata set corresponding to the target bucket, the process also includes: The signature authentication information is verified to verify the user identity of the client. If the verification is successful, the step of querying file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket is triggered. If the verification fails, the task request for the query task is rejected, and the client is provided with corresponding indication information indicating that the signature verification failed.

6. The method according to claim 1, characterized in that, After querying the metadata of at least one target file from the metadata set corresponding to the target bucket to obtain the metadata of the file that matches the file constraint information, the process further includes: Cache the metadata of at least one of the target files; Upon receiving a request to export metadata, a metadata file including the metadata of the at least one target file is generated based on the cached metadata of the at least one target file, and the metadata file is transmitted to the client so that the client can export the metadata file; And / or, upon detecting that the client submits the task request for the query task again, the task request is responded to based on the cached metadata of the at least one target file.

7. The method according to claim 6, characterized in that, The object storage server is deployed on a cloud platform, and a console is also built on the cloud platform as the entry point for the query interface. The console is used to: provide a visual graphical interface to client users, and to call the query interface according to the task request of the obtained query task, and to generate a metadata file based on the corresponding cached metadata and transmit it to the client according to the obtained export request for metadata. The process of generating a metadata file including the metadata of the at least one target file based on the cached metadata of the at least one target file includes: The console loads the cached metadata of the at least one target file and generates a metadata file including the metadata of the at least one target file according to a predetermined file format. The response processing for the task request based on the metadata of the at least one target file cached includes: The console queries the metadata of the at least one target file in the cache, and processes the task request in response based on the queried metadata of the at least one target file.

8. The method according to claim 7, characterized in that, The console is also used to: provide the data owner with the function of authorizing the data owner's interface operation permissions for the query interface to the target user through the visual graphical interface; The target user is either a sub-user of the data owner or another user distinct from the data owner.

9. A query and statistics device for object storage information, characterized in that, include: The acquisition module is used to acquire the task request of the query task submitted by the client through the query interface pre-set on the object storage service side; the request parameters of the task request include at least the file constraint information corresponding to the target query type and the bucket information of the target storage bucket; The query module is used to query file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket, and obtain the metadata of at least one target file; The statistics module is used to perform statistical processing on the metadata of the at least one target file to obtain statistical results; The response processing module is used to generate response information including the statistical results and to send the response information back to the client through the query interface.

10. A service platform, characterized in that, include: The object storage server is equipped with a query interface. The object storage server is used for: The task request submitted by the client is obtained through the query interface pre-set on the object storage service side; the request parameters of the task request include at least file constraint information corresponding to the target query type and bucket information of the target storage bucket; Query the file metadata that matches the file constraint information from the metadata set corresponding to the target storage bucket to obtain the metadata of at least one target file; Perform statistical processing on the metadata of the at least one target file to obtain statistical results; Generate response information including the statistical results, and send the response information back to the client through the query interface.

11. The service platform according to claim 10, characterized in that, The service platform also includes a console as the entry point for the query interface; The console is used to: provide a visual graphical interface to client users, and to invoke the query interface based on the task request of the obtained query task.

12. The service platform according to claim 11, characterized in that, The object storage server is also used for: Cache the metadata of at least one of the target files; The console is also used for: Upon receiving a request to export metadata, a metadata file including the metadata of the at least one target file is generated based on the cached metadata of the at least one target file, and the metadata file is transmitted to the client so that the client can export the metadata file; And / or, upon detecting that the client submits the task request for the query task again, the task request is responded to based on the cached metadata of the at least one target file.

13. The service platform according to claim 11, characterized in that, The console is also used for: The visual graphical interface provides the data owner with the function of authorizing the target user to access the query interface. The target user is either a sub-user of the data owner or another user distinct from the data owner.

14. A query and statistics system for object storage information, characterized in that, include: At least one client; The service platform as described in any one of claims 10-13; The service platform provides the client with query and statistics services for object storage information, at least through the server that has the query interface.

Images