Object metadata storage method and device, equipment, medium and program product

Abstract

The invention provides an object metadata storage method and device, equipment, a medium and a program product, and the method comprises the steps: caching metadata index information of an object after the object is stored in an object storage system, publishing the metadata index information to a message queue as a message, and determining that the storage of the object is completed after the message is successfully published; a batch processing request is sent to an ES cluster, so that the multiple pieces of metadata index information are stored in the ES cluster in a batch mode; the step of caching the metadata index information comprises the substeps of distributing a unique identifier for the metadata index information, performing associated caching on the metadata index information and the unique identifier, and caching the unique identifier into an inverted list corresponding to each attribute value; and when metadata retrieval triggering is received, performing retrieval in the cached metadata index information and the ES cluster by utilizing the obtained retrieval condition, and obtaining a target object matched with the retrieval condition based on a retrieval result. According to the method, metadata batch storage is realized, and the object storage performance is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of object storage technology, and in particular to an object metadata storage method, device, equipment, medium and program product. Background Art

[0002] In object storage systems, metadata retrieval is an important function that enables users to quickly locate and access required object data by querying metadata. The implementation of metadata retrieval is generally to establish an Elaticsearch (ES) cluster outside the object storage system to store part of the object metadata. When retrieval is required, the corresponding metadata is queried from the ES cluster.

[0003] However, the current object metadata storage process has some technical difficulties. First, since the ES cluster is an independent component, it requires additional resources and management costs to maintain and operate. Under cost constraints, the performance of the ES cluster is often not as good as the object storage system, which causes the object write speed to be limited by the metadata write speed, which in turn affects the write performance of the object storage system. Secondly, in the current object metadata storage process, each object can only generate one metadata write request, which requires sending a large number of requests to the ES cluster when facing the concurrent storage of massive objects, greatly reducing the processing speed of the ES cluster. Summary of the invention

[0004] To overcome the problems existing in the related art, the present application provides an object metadata storage method, apparatus, device, medium and program product.

[0005] According to a first aspect of an embodiment of the present application, a method for storing object metadata is provided, including:

[0006] After the object is stored in the object storage system, the metadata index information of the object is cached, and the metadata index information of the object is published as a message to a message queue, and after the message is successfully published, it is determined that the storage of the object is completed;

[0007] Sending a batch processing request to the ES cluster to batch store the metadata index information in the plurality of messages into the ES cluster;

[0008] The metadata index information of the object includes a plurality of attribute values ​​indicating the attribute information of the object; caching the metadata index information of the object specifically includes: allocating a unique identifier for the metadata index information of the object, caching the metadata index information of the object in association with the unique identifier, and caching the unique identifier in an inverted list corresponding to each of the attribute values; the inverted list is used to quickly retrieve the metadata index information of the object based on the attribute value;

[0009] When a metadata search request is received, the cached metadata index information and the ES cluster are searched using the obtained search conditions, and a target object matching the search conditions is acquired based on the search results.

[0010] According to a second aspect of an embodiment of the present application, there is provided an object metadata storage device, including:

[0011] The metadata index information caching and message publishing module is used to cache the metadata index information of the object after the object is stored in the object storage system, and publish the metadata index information of the object as a message to the message queue, and after the message is successfully published, determine that the storage of the object is completed;

[0012] A metadata index information batch storage module, used to send a batch processing request to the ES cluster to batch store the metadata index information in the plurality of messages into the ES cluster;

[0013] The metadata index information of the object includes a plurality of attribute values ​​indicating the attribute information of the object; caching the metadata index information of the object specifically includes: allocating a unique identifier for the metadata index information of the object, caching the metadata index information of the object in association with the unique identifier, and caching the unique identifier in an inverted list corresponding to each of the attribute values; the inverted list is used to quickly retrieve the metadata index information of the object based on the attribute value;

[0014] When a metadata search request is received, the cached metadata index information and the ES cluster are searched using the obtained search conditions, and a target object matching the search conditions is acquired based on the search results.

[0015] According to a third aspect of an embodiment of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0016] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.

[0017] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0018] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0019] The embodiment of the present application, by introducing a message queue, encapsulates the metadata index information of the object into a message and publishes it to the message queue, so that the object storage process can be considered completed after the message is published, without having to continue waiting for the corresponding metadata index information to be actually stored in the ES cluster, realizing asynchronous processing of the object storage process and the metadata storage process, effectively decoupling the object storage process and the metadata index information storage process, avoiding the problem that the object storage speed is limited by the metadata storage speed, and significantly improving the storage performance of the object storage system. At the same time, the embodiment of the present application, through a batch processing request, realizes batch storage of metadata index information, effectively reduces frequent requests to the ES cluster, optimizes the use of network and storage resources, and improves the storage efficiency of the ES cluster.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0022] Figure 1 It is a flowchart of an object metadata storage method shown in the present application according to an exemplary embodiment.

[0023] Figure 2 It is a schematic diagram of an inverted index of a string type attribute according to an exemplary embodiment of the present application.

[0024] Figure 3 It is a flowchart of another object metadata storage method shown in the present application according to an exemplary embodiment.

[0025] Figure 4 It is a hardware structure diagram of a computer device where an object metadata storage device is located according to an exemplary embodiment of the present application.

[0026] Figure 5 It is a structural block diagram of an object metadata storage device shown in the present application according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0028] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0030] Before discussing the present application scheme in depth, it is necessary to briefly explain some basic concepts involved in the scheme in order to better understand the content of the present application.

[0031] Object: It is the basic unit of storage in object storage. An object can be a collection of file data and its related attribute information, including the object's unique identifier, object data, and object metadata.

[0032] Object metadata: It is the attribute information related to the object, usually including the object name, access rights, creation time, size, etc., which is used to describe the data characteristics and storage characteristics of the object.

[0033] Metadata index information: It is the key attribute information in object metadata that is often used for object retrieval, such as object name, creation date, size, etc.

[0034] Inverted index: Also known as reverse index, it is a storage format that implements the term-document matrix. Its main idea is to quickly obtain a list of documents containing the term based on the term.

[0035] Inverted list: It is the core data structure of the inverted index. It is based on terms and maps each term to a list of documents containing the term. This list of documents is also called an inverted list. For each term, there is an inverted list that stores the document ID corresponding to it.

[0036] Term table: It is a data structure used to store terms in the inverted index.

[0037] Document identification table: a data structure used to store the associated mapping relationship between document ID and actual document.

[0038] Dictionary tree: It is a multi-branch tree structure used for fast retrieval. It regards a string as a character sequence and constructs a tree structure from top to bottom according to the order of the character sequence, where each edge of the tree structure corresponds to a character.

[0039] After the metadata retrieval function is enabled in the current object storage system, the object metadata storage process is roughly as follows:

[0040] Store objects in the object storage system and generate object metadata;

[0041] Extract necessary metadata index information from the generated object metadata and encapsulate the index information into a whole;

[0042] Constructing a metadata write request according to the encapsulated index information to store all the index information as a document;

[0043] Send the metadata write request to the ES cluster to complete the storage of a single object metadata index document.

[0044] In the current object metadata storage process, object metadata storage and object storage are performed synchronously, which means that each time the object storage system stores an object, it must wait for the corresponding metadata to be written to the ES cluster before it can process the storage of subsequent objects. In the case where the performance of the ES cluster is not as good as that of the object storage system, this storage method will cause the object storage speed to be limited by the metadata storage speed, thereby affecting the write performance of the object storage system. In addition, when writing object metadata to the ES cluster, only one metadata write request can be generated for each object, resulting in a large number of requests to be sent to the ES cluster when facing the concurrent storage of massive objects, which will greatly reduce the processing speed of the ES cluster.

[0045] In order to solve the above problems, the present application provides a new object metadata storage method. This method introduces a message queue, encapsulates the metadata index information of the object into a message and publishes it to the message queue, so that the object storage process can be considered completed after the message is published, without having to continue waiting for the corresponding metadata index information to be actually stored in the ES cluster, thereby realizing asynchronous processing of the object storage process and the metadata storage process, effectively decoupling the object storage process and the metadata index information storage process, avoiding the problem that the object storage speed is limited by the metadata storage speed, and significantly improving the storage performance of the object storage system. At the same time, through a batch processing request, batch storage of metadata index information is realized, which effectively reduces frequent requests to the ES cluster, optimizes the use of network and storage resources, and improves the storage efficiency of the ES cluster.

[0046] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0047] Figure 1 FIG. 1 is a flowchart of a method for storing object metadata according to an exemplary embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0048] S101, after the object is stored in the object storage system, the metadata index information of the object is cached, and the metadata index information of the object is published as a message to the message queue, and after the message is successfully published, it is determined that the storage of the object is completed;

[0049] S102: Send a batch processing request to the ES cluster to batch store metadata index information in multiple messages into the ES cluster.

[0050] In step S101, by introducing a message queue, the metadata index information of the object is encapsulated into a message and published to the message queue, so that the object storage process can be considered completed after the message is published, without having to continue waiting for the corresponding metadata index information to be actually stored in the ES cluster. This method realizes the asynchronous processing of the object storage process and the metadata storage process, effectively decouples the object storage process and the metadata index information storage process, avoids the problem that the object storage speed is limited by the metadata storage speed, and significantly improves the storage performance of the object storage system. Among them, the message queue can be any message queue system that can effectively handle high-concurrency message publishing and consumption, such as RabbitMQ, Kafka or ActiveMQ, etc., and this application does not limit this.

[0051] In step S102, when a certain number of messages are accumulated in the message queue, or a predetermined time interval is reached, a specified number of messages can be obtained from the message queue, and the metadata index information in these messages can be sent to the ES cluster for batch storage at one time through a batch processing request, thereby effectively reducing frequent requests to the ES cluster, optimizing the use of network and storage resources, and improving the storage efficiency of the ES cluster. Among them, the number of batch processed messages can be adjusted according to the actual needs of the user, and this application does not impose any restrictions on this.

[0052] During the specific implementation process, in order to improve the efficiency and response speed of message processing, a new thread can be enabled in the object storage system to process messages in the message queue. This new thread can continuously obtain a specified number of messages in the message queue through the consumer mode corresponding to the message queue (such as the basicGet mode corresponding to RabbitMQ), and parse the metadata index information contained in the message until the metadata index information in the message is stored in the ES cluster and the message consumption is considered complete. In the ES cluster, batch storage operations can be implemented through the BulkProcessor object. The BulkProcessor object is a tool provided by the ES cluster for batch processing requests. It can create a bulkRequest object, generate an IndexRequest operation instance for the metadata index information of each object, and add these IndexRequest operation instances to the bulkRequest object. IndexRequest is a class in the ES cluster client, which is used to add or update index documents to the index structure of the ES cluster. By calling the flush() method in the BulkProcessor object, all operation instances in the bulkRequest object can be sent to the ES cluster through a batch processing request to implement batch writing of metadata index information.

[0053] In the above embodiment, by introducing the message queue, asynchronous processing of the object storage process and the metadata index information storage process is realized, which effectively improves the performance of the object storage system and the storage efficiency of the ES cluster. However, during the asynchronous processing, there is a special case: the object has been successfully stored in the object storage system, but the message corresponding to its metadata index information has not been consumed, so that the metadata index information has not been actually stored in the ES cluster. This situation will cause the user to be unable to retrieve the corresponding metadata information from the ES cluster when using the object metadata retrieval function, and thus cannot realize object retrieval.

[0054] To solve this problem, the present embodiment may also adopt a pre-caching strategy in step S101, that is, before publishing the metadata index information of the object as a message to the message queue, the information is cached in advance. In this way, when a metadata retrieval request is received, the obtained retrieval conditions can be used to search the cached metadata index information and the ES cluster at the same time, and the target object matching the retrieval conditions can be obtained based on the retrieval results. In this way, even if there are messages corresponding to some metadata index information in the message queue that have not been consumed, the user can still retrieve the corresponding metadata information in the cache, thereby ensuring the integrity and reliability of the retrieval function.

[0055] In the specific retrieval process, the obtained retrieval conditions can be used to search the cached metadata index information and the ES cluster respectively to obtain the first retrieval result and the second retrieval result. Then, the first retrieval result and the second retrieval result are processed as a union to obtain the target metadata index information that matches the retrieval conditions. Finally, based on the target metadata index information, the target object that matches the retrieval conditions is obtained from the object storage system.

[0056] To implement the pre-caching strategy, an independent object metadata retrieval module can be set up inside the object storage system, and the module can be used as a cache space to pre-cache the metadata index information of the object.

[0057] Considering that the metadata index information of an object includes several attribute values ​​indicating the attribute information of the object, and users usually use these attribute values ​​as retrieval conditions when performing metadata retrieval, the object metadata retrieval module can construct an inverted index for each attribute information involved in the metadata when caching the metadata index information, so as to realize fast retrieval of the metadata index information based on the attribute values.

[0058] In the process of building an inverted index, it is also considered that there may be multiple attributes of different data types in the metadata attributes, so the inverted index can also be built in different ways according to the different types of attributes. For example, for metadata attributes of string type (such as BucketName, ObjName, non-numeric attributes in userMeta, contentType, accountId, eTag, etc.), the non-repeated attribute values ​​that appear in each attribute can be used as terms, and the metadata index information identifier containing the term is saved in the inverted list corresponding to the term. These inverted lists use the term as the key and the mapped inverted list as the value, and are saved in the inverted list of the TreeMap structure corresponding to different attributes. The term elements in the inverted list of the string type attribute are arranged in dictionary order, and the term is added to the term table of the TreeSet structure. When retrieving metadata index information, the dictionary tree can be combined to quickly obtain the position of the term in the term table with the prefix specified in the search condition to achieve fuzzy search, or the term can be directly searched accurately through the inverted table. Figure 2 A schematic diagram of an inverted index of a string type attribute is shown.

[0059] For metadata attributes of numeric type and date type (such as objSize, numeric attributes in userMeta, and lastModified, etc.), the non-repeated attribute values ​​that appear in each attribute can also be used as terms (date type attribute values ​​must first be converted to millisecond timestamp values), and the inverted table and term table are generated in a similar way to building an inverted index for string type attributes. The term elements in the inverted table of numeric type and date type attributes are arranged according to the size of the values. When retrieving metadata index information, you can directly perform an accurate term search, or perform a range fuzzy search based on the terms within the search condition matching range.

[0060] When caching the metadata index information of an object, a unique identifier may be assigned to the metadata index information of the object, the metadata index information of the object and the unique identifier may be associated and cached, and the unique identifier may be cached in an inverted list corresponding to each attribute value included in the metadata index information.

[0061] During the specific implementation process, a document ID can be first assigned to the metadata index information of the object as a unique identifier, and the metadata index information and the document ID can be associated and stored in the document identification table in the memory of the object metadata retrieval module. This document identification table can provide a mapping relationship to associate the document ID with the metadata index information, so that the corresponding metadata index information can be quickly retrieved according to the document ID later. Then, the metadata index information content is parsed, the attributes in the metadata index information are traversed, and the inverted list corresponding to the attribute value included in the metadata index information is found in the inverted list corresponding to each attribute, and the document ID associated with this metadata index information is saved in the inverted list corresponding to these attribute values; new attribute values ​​that appear need to add new records in the inverted list and the term table, and the attribute values ​​of string type need to add the location information of the attribute value in the term table to the corresponding node of the dictionary tree. Among them, the specific form of the unique identifier depends on the actual application scenario and requirements, and this application does not limit this.

[0062] The caching process of the above metadata index information is described with a specific example:

[0063] Assume that the metadata index information of an object includes an attribute value "Ab" indicating the object name, and the document ID assigned to the metadata index information is "2". When caching, first search for the inverted list corresponding to the attribute value "Ab" in the inverted list of the object name attribute cached in the object metadata retrieval module. If the inverted list corresponding to the attribute value "Ab" is found to be [6], then the document ID "2" of the current metadata index information is added to the inverted list corresponding to the attribute value "Ab", making it [2, 6]. If the attribute value "Ab" is a newly appearing attribute value in the object name attribute cached in the object metadata retrieval module, then add the new term "Ab" to the term table of the object name attribute, generate the inverted list [2] corresponding to the new term "Ab", and determine the position of the new term "Ab" in the term table according to the dictionary order.

[0064] When storing metadata index information in batches to the ES cluster, metadata index information storage may fail due to network fluctuations, excessive load on the ES cluster, or other reasons. To ensure that all metadata index information can be successfully stored in the ES cluster, the metadata index information that failed to be stored needs to be republished as a message to the message queue so that subsequent attempts can be made to store the metadata index information in the ES cluster.

[0065] In the specific implementation process, the execution results of the IndexRequest operation instances corresponding to the batch processing request and each metadata index information can be analyzed to identify whether the metadata index information fails to be stored. There are two situations in which the metadata index information fails to be stored: one is that all the metadata index information stored in the batch fails to be stored, and the other is that only part of the metadata index information fails to be stored.

[0066] If all the metadata index information stored in batches fails to be stored, it may be manifested as an exception in the batch processing request sent to the ES cluster, resulting in the request failure. In this case, all the metadata index information processed this time should be re-published as a message to the message queue so that the storage operation can be performed again after the network is stable or the ES cluster load is reduced.

[0067] If some metadata index information fails to be stored, it may be manifested as the execution failure of the IndexRequest operation instance corresponding to some metadata index information. In this case, you only need to republish the metadata index information corresponding to the failed operation instance as a message to the message queue. In this way, only the failed part will be republished, and the successful part does not need to be processed repeatedly, which can improve storage efficiency and reduce unnecessary resource consumption.

[0068] In this way, it can be ensured that even in the face of storage failure, the metadata index information can be successfully stored in the ES cluster, ensuring the stability of the system and the integrity of the data.

[0069] For metadata index information that has been successfully stored in the ES cluster, the cache corresponding to this metadata index information can be removed. Because this metadata index information has been stored in the ES cluster, subsequent retrieval can directly obtain this metadata index information from the ES cluster without continuing to occupy cache space, thereby reducing unnecessary data redundancy and optimizing resource utilization.

[0070] When removing the cache of successfully stored metadata index information, you can first determine the target identifier associated with these successfully stored metadata index information, then determine the target inverted list including the target identifier, remove the target identifier in the target inverted list, and finally remove the associated cache of these successfully stored metadata index information and the target identifier.

[0071] In the specific implementation process, first, based on the successfully stored metadata index information, the document ID associated with the metadata index information can be accurately retrieved from the document identification table of the object metadata retrieval module. Next, the inverted table in the object metadata retrieval module is traversed to find all inverted lists containing the document ID, and the document IDs in these inverted lists are removed. Finally, the record of the document ID in the document identification table is deleted to complete the cache removal process. This series of operations ensures that only the cache of metadata index information successfully stored in the ES cluster is removed, while the information that has not been successfully stored remains in the cache for subsequent retries.

[0072] Based on the above embodiments, Figure 3 FIG. 1 is a flowchart of another object metadata storage method according to an exemplary embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0073] S301, after the object is stored in the object storage system, the metadata index information of the object is cached, and the metadata index information of the object is published as a message to the message queue, and after the message is successfully published, it is determined that the storage of the object is completed;

[0074] S302, sending a batch processing request to the ES cluster to batch store metadata index information in multiple messages into the ES cluster;

[0075] S303: Determine whether the metadata index information is successfully stored in the ES cluster. If yes, execute step S304; if no, execute step S305.

[0076] S304, remove the cache corresponding to the metadata index information successfully stored in the ES cluster;

[0077] S305: Publish the metadata index information that has not been successfully stored in the ES cluster as a message to the message queue again.

[0078] The specific implementation process of each step in this embodiment can be found in the description of the above embodiment, which will not be described in detail.

[0079] In addition, in the process of caching the metadata index information of the object, you may encounter special situations such as abnormal power failure, which may cause the cached content in the object metadata retrieval module to be lost. If appropriate measures are not taken, the loss of cache will hinder the function of the caching step, that is, to solve the problem that when the message in the message queue has not been consumed and the metadata index information has not been actually stored in the ES cluster, the user cannot retrieve the corresponding metadata information from the ES cluster when using the object metadata retrieval function.

[0080] To deal with such situations and ensure that the cache function of the object metadata retrieval module can be restored in time, when the metadata index information cache of an object is lost, the metadata index information in the message queue that is not stored in the ES cluster can be re-cached. This measure ensures that even in abnormal situations, the complete retrieval function can be maintained, ensuring that users can retrieve the required metadata information.

[0081] In the specific implementation process, the state of the object metadata retrieval module can be continuously monitored. Once the cache loss is detected, the recovery process is triggered to extract the metadata index information that has not been successfully stored in the ES cluster from the message queue and reload it into the cache of the object metadata retrieval module. Through this recovery mechanism, the fault tolerance capability for abnormal situations can be improved, the integrity of data and the continuity of retrieval can be ensured, and the stability and reliability of object metadata storage can be enhanced.

[0082] Corresponding to the above-mentioned embodiment of the object metadata storage method, the present application also provides an embodiment of an object metadata storage device and a terminal used therein.

[0083] The embodiment of the object metadata storage device of the present application can be applied to a computer device, such as a server or a terminal device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by its processor reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 4 FIG. 1 is a hardware structure diagram of a computer device where the object metadata storage device of the embodiment of the present application is located, except Figure 4In addition to the processor 401, memory 402, network interface 403, and non-volatile memory 404 shown, the server or electronic device where the object metadata storage device is located in the embodiment may also include other hardware, usually according to the actual function of the computer device, which will not be described in detail.

[0084] Figure 5 1 is a structural block diagram of an object metadata storage device according to an exemplary embodiment of the present application. Figure 5 As shown, the device comprises:

[0085] The metadata index information caching and message publishing module 501 is used to cache the metadata index information of the object after the object is stored in the object storage system, and publish the metadata index information of the object as a message to the message queue, and after the message is successfully published, determine that the storage of the object is completed;

[0086] The metadata index information batch storage module 502 is used to send a batch processing request to the ES cluster to batch store the metadata index information in the plurality of messages into the ES cluster;

[0087] The metadata index information of the object includes a plurality of attribute values ​​indicating the attribute information of the object; caching the metadata index information of the object specifically includes: allocating a unique identifier to the metadata index information of the object, caching the metadata index information of the object in association with the unique identifier, and caching the unique identifier in an inverted list corresponding to each attribute value; the inverted list is used to quickly retrieve the metadata index information of the object based on the attribute value;

[0088] When a metadata search trigger is received, the cached metadata index information and the ES cluster are searched using the obtained search conditions, and the target object matching the search conditions is obtained based on the search results.

[0089] Correspondingly, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the object metadata storage method described in any of the above embodiments are implemented.

[0090] Accordingly, the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the object metadata storage method described in any of the above embodiments.

[0091] Accordingly, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the object metadata storage method recorded in any of the above embodiments.

[0092] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0093] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0094] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0096] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0097] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for storing object metadata, characterized in that: include: After the object is stored in the object storage system, the metadata index information of the object is cached, and the metadata index information of the object is published as a message to a message queue, and after the message is successfully published, it is determined that the storage of the object is completed; Sending a batch processing request to the ES cluster to batch store the metadata index information in the plurality of messages into the ES cluster; The metadata index information of the object includes a plurality of attribute values ​​indicating the attribute information of the object; caching the metadata index information of the object specifically includes: allocating a unique identifier for the metadata index information of the object, caching the metadata index information of the object in association with the unique identifier, and caching the unique identifier in an inverted list corresponding to each of the attribute values; the inverted list is used to quickly retrieve the metadata index information of the object based on the attribute value; When a metadata search request is received, the cached metadata index information and the ES cluster are searched using the obtained search conditions, and a target object matching the search conditions is acquired based on the search results.

2. The method according to claim 1, characterized in that Also includes: If there is metadata index information that has not been successfully stored in the ES cluster, the metadata index information that has not been successfully stored in the ES cluster is republished as a message to the message queue.

3. The method according to claim 1, characterized in that Also includes: The cache corresponding to the metadata index information successfully stored in the ES cluster is removed.

4. The method according to claim 3, characterized in that Removing the cache corresponding to the metadata index information successfully stored in the ES cluster, including: Determine a target identifier associated with the metadata index information successfully stored in the ES cluster; Determine a target posting list including the target identifier, and remove the target identifier from the target posting list; The associated cache of the metadata index information successfully stored in the ES cluster and the target identifier is removed.

5. The method according to claim 1, characterized in that Using the obtained search condition to search in the cached metadata index information and the ES cluster, and obtaining the target object matching the search condition based on the search result, including: Using the obtained search condition, searching in the cached metadata index information and the ES cluster respectively, to obtain a first search result and a second search result; Performing a union process on the first search result and the second search result to obtain target metadata index information matching the search condition; Based on the target metadata index information, a target object matching the search condition is acquired from the object storage system.

6. The method according to claim 1, characterized in that Also includes: If the metadata index information cache of an object is lost, the metadata index information in the message queue that is not stored in the ES cluster is cached again.

7. An object metadata storage device, characterized in that: include: The metadata index information cache and message publishing module is used to cache the metadata index information of the object after the object is stored in the object storage system, and publish the metadata index information of the object as a message to the message queue, and after the message is successfully published, determine that the storage of the object is completed; A metadata index information batch storage module, used to send a batch processing request to the ES cluster to batch store the metadata index information in the plurality of messages into the ES cluster; The metadata index information of the object includes a plurality of attribute values ​​indicating the attribute information of the object; caching the metadata index information of the object specifically includes: allocating a unique identifier for the metadata index information of the object, caching the metadata index information of the object in association with the unique identifier, and caching the unique identifier in an inverted list corresponding to each of the attribute values; the inverted list is used to quickly retrieve the metadata index information of the object based on the attribute value; When a metadata search request is received, the cached metadata index information and the ES cluster are searched using the obtained search conditions, and a target object matching the search conditions is acquired based on the search results.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images