Metadata query method and device, equipment, medium and product

Abstract

The invention relates to a metadata query method and device, equipment, a medium and a product, and the method comprises the steps: determining the identification information of metadata of a target bucket corresponding to a search request and the number of target node equipment in a distributed system under the condition that the search request is received; the search request is a request for searching target object data in a target bucket; decomposing the search request into a plurality of sub-requests according to the identification information and the number of the target node devices, and determining a metadata processing range corresponding to each sub-request; respectively distributing each sub-request and the metadata processing range corresponding to each sub-request to the corresponding target node equipment, so that each target node equipment processes the corresponding metadata; and summarizing the processing results of the node devices to obtain a search result corresponding to the search request. Therefore, the search request is decomposed into the plurality of sub-requests, so that the plurality of target node devices process the corresponding sub-requests in parallel, and the response efficiency of the search operation can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a metadata query method, apparatus, device, medium and product. Background Technology

[0002] In a distributed storage system, multiple buckets can be created, each independently holding data of a specific type, business function, or user. When performing a lookup operation on an object within a bucket, it is necessary to traverse the massive metadata set within the bucket.

[0003] Because traversing the metadata collection in the bucket is time-consuming and requires fine filtering and sorting of data in memory, it places extremely high demands on memory resources, resulting in low response efficiency for object lookup operations. Summary of the Invention

[0004] Therefore, it is necessary to provide a metadata query method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a metadata query method, which includes:

[0006] Upon receiving a lookup request, determine the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system; the lookup request is a request used to find target object data within the target bucket.

[0007] Based on the identification information and the number of target node devices, the lookup request is broken down into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

[0008] Each sub-request and its corresponding metadata processing scope are allocated to the corresponding target node devices so that each target node device can process the corresponding metadata.

[0009] The processing results of each node device are summarized to obtain the search result corresponding to the search request.

[0010] In one embodiment, based on the identification information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined, including:

[0011] Based on the identification information, determine the index fragmentation information of the metadata;

[0012] Based on the index sharding information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

[0013] In one embodiment, the index sharding information includes the number of index shards and the index sharding range; based on the index sharding information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing range corresponding to each sub-request is determined, including:

[0014] The search request is broken down into multiple sub-requests based on the number of target node devices; the number of sub-requests is determined based on the number of target node devices.

[0015] The number of index units processed by each target node device is determined based on the number of index shards and the number of target node devices.

[0016] The index sharding range corresponding to each sub-request is determined according to the order of the sub-requests, the index sharding range, and the number of unit indexes.

[0017] In one embodiment, the method further includes: identifying idle node devices in the distributed system as target node devices.

[0018] In one embodiment, the processing results of each node device are aggregated to obtain the search result corresponding to the search request, including: performing data filtering and sorting processing on the processing results of each node device to obtain the search result.

[0019] In one embodiment, the processing results of each node device are filtered and sorted to obtain search results, including: filtering and sorting the processing results of each node device to obtain sorted processing results; and determining a preset number of processing results from the sorted processing results as search results.

[0020] Secondly, this application also provides a metadata query device, which includes:

[0021] The lookup request receiving module is used to determine the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system when a lookup request is received; the lookup request is a request to find the target object data in the target bucket.

[0022] The lookup request decomposition module is used to decompose the lookup request into multiple sub-requests based on the identification information and the number of target node devices, and to determine the metadata processing scope corresponding to each sub-request.

[0023] The metadata processing module is used to allocate each sub-request and the metadata processing scope corresponding to each sub-request to the corresponding target node devices, so that each target node device can process the corresponding metadata.

[0024] The search result determination module is used to summarize the processing results of each node device and obtain the search result corresponding to the search request.

[0025] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0026] Upon receiving a lookup request, determine the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system; the lookup request is a request used to find target object data within the target bucket.

[0027] Based on the identification information and the number of target node devices, the lookup request is broken down into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

[0028] Each sub-request and its corresponding metadata processing scope are allocated to the corresponding target node devices so that each target node device can process the corresponding metadata.

[0029] The processing results of each node device are summarized to obtain the search result corresponding to the search request.

[0030] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0031] Upon receiving a lookup request, determine the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system; the lookup request is a request used to find target object data within the target bucket.

[0032] Based on the identification information and the number of target node devices, the lookup request is broken down into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

[0033] Each sub-request and its corresponding metadata processing scope are allocated to the corresponding target node devices so that each target node device can process the corresponding metadata.

[0034] The processing results of each node device are summarized to obtain the search result corresponding to the search request.

[0035] Fifthly, this application also provides a computer program product comprising a computer program that, when executed by a processor, performs the following steps:

[0036] Upon receiving a lookup request, determine the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system; the lookup request is a request used to find target object data within the target bucket.

[0037] Based on the identification information and the number of target node devices, the lookup request is broken down into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

[0038] Each sub-request and its corresponding metadata processing scope are allocated to the corresponding target node devices so that each target node device can process the corresponding metadata.

[0039] The processing results of each node device are summarized to obtain the search result corresponding to the search request.

[0040] The aforementioned metadata query method, apparatus, device, medium, and product, upon receiving a search request, determine the identifier information of the metadata of the target bucket corresponding to the search request and the number of target node devices in the distributed system; the search request is a request to search for target object data within the target bucket; based on the identifier information and the number of target node devices, the search request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined; each sub-request and its corresponding metadata processing scope are allocated to the corresponding target node devices, so that each target node device processes the corresponding metadata; the processing results of each node device are aggregated to obtain the search result corresponding to the search request. This application employs the above method, decomposing the search request into multiple sub-requests, allowing multiple target node devices to process the corresponding sub-requests in parallel, thereby significantly shortening the processing time and improving the response efficiency of the search operation; moreover, it can fully utilize the computing resources of multiple node devices in the distributed system, making system resources more effectively utilized. Attached Figure Description

[0041] Figure 1 This application provides an application environment diagram for the metadata query method provided in some embodiments of this application;

[0042] Figure 2 A schematic diagram illustrating the interaction relationship between the master control node, the client, and each target node device provided in some embodiments of this application;

[0043] Figure 3 A schematic diagram illustrating the aggregation of processing results from target node devices by the master control node, as provided in some embodiments of this application;

[0044] Figure 4 Flowcharts of metadata query methods provided in some embodiments of this application;

[0045] Figure 5 A flowchart for determining the metadata processing scope corresponding to each sub-request, provided for some embodiments of this application;

[0046] Figure 6A flowchart for determining the index fragment range corresponding to each sub-request is provided for some embodiments of this application;

[0047] Figure 7 A flowchart for determining search results provided in some embodiments of this application;

[0048] Figure 8 Structural block diagram of a metadata query apparatus provided in some embodiments of this application;

[0049] Figure 9 This is an internal structural diagram of a computer device provided in some embodiments of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] The metadata query method provided in this application embodiment can be applied to, for example... Figure 1 The distributed system architecture shown includes clients and a distributed storage system based on a Ceph-based security video cloud storage platform. Clients are users or applications that utilize the storage service; they are also the initiators of system requests and the recipients of the final results. The distributed storage system of the Ceph-based security video cloud storage platform comprises multiple L_RGW gateway nodes and a storage cluster for storing and managing massive amounts of security video data. Multiple L_RGW gateway nodes constitute the List gateway layer, and the storage cluster constitutes the metadata storage layer. Both the List gateway layer and the metadata storage layer run on servers. The List gateway layer is located between the clients and the metadata storage layer. In the List gateway layer, the L_RGW node that receives standard S3 / Swift ListObjects requests from the clients is the master node, and the other L_RGW nodes are compute nodes.

[0052] like Figure 2 and Figure 3 As shown, when the master node receives a lookup request from a client, it first determines the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system; the lookup request is a request to find target object data within the target bucket; then, based on the identifier information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined, for example... Figure 2The metadata processing range for the sub-request corresponding to node L_RGW2 is "list_range(k, 2k-1)"; then, each sub-request and its corresponding metadata processing range are allocated to the corresponding target node devices, i.e., to the corresponding compute nodes, so that each target node device can process the corresponding metadata; finally, the processing results of each node device are summarized to obtain the search result corresponding to the search request, for example, Figure 2 and Figure 3 Each node device in the system returns its processing result to the master node, which then returns the final search result to the client. In this way, by decomposing the search request into multiple sub-requests, allowing multiple target node devices to process the corresponding sub-requests in parallel, the processing time can be greatly shortened, thereby improving the response efficiency of the search operation. Furthermore, the computing resources of multiple node devices in the distributed system can be fully utilized, making system resources more effectively used.

[0053] In one embodiment, such as Figure 4 As shown, this method is illustrated using the aforementioned server as an example. It can be understood that this method can also be applied to systems including terminals and servers, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] Step 402: Upon receiving a search request, determine the identifier information of the metadata of the target bucket corresponding to the search request and the number of target node devices in the distributed system.

[0055] A lookup request is a request used to find data about a target object within a target bucket. For example, a query request for "all video clips from camera 123 in area A between 9:00 AM and 11:00 AM on November 25, 2025" corresponds to the ListObjects operation in S3, with a prefix (A / 123 / 2025-11-25 / ) and a time range filter. A target bucket is a bucket that stores data about the target object. For example, in the above example, the target bucket is named "city-surveillance," and video clips generated daily by several cameras within the target area can be stored in this bucket according to the structure "area / camera ID / date." Metadata identification information refers to metadata index shard information related to the target bucket, such as the number of metadata index shards in the target bucket.

[0056] The distributed system in this embodiment is a Ceph-based security video cloud storage platform, which includes multiple L_RGW gateway nodes and a storage cluster for storing and managing massive amounts of security video data. The target node devices in the distributed system are those capable of processing search requests, i.e., the aforementioned compute nodes. For example, the entire cluster has five L_RGW nodes available for scheduling, with one L_RGW serving as the master node (receiving client requests) and the remaining four as compute nodes.

[0057] Optionally, when the master node receives a lookup request from the client, it first obtains the number of metadata index shards of the target bucket through the metadata management module of the query system, and uses the number of metadata index shards of the target bucket as the metadata identification information; at the same time, it obtains the number of target node devices available for scheduling in the current distributed system through the resource management module of the system.

[0058] Step 404: Based on the identification information and the number of target node devices, the search request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

[0059] The metadata processing range for each sub-request is the range of metadata index shards that each sub-request needs to process. Optionally, assuming the master node calculates that each node should process 20 shards, it assigns a different shard range to each node. For example, node 1 processes shards 0-19, node 2 processes shards 20-39, and so on.

[0060] Optionally, the number of sub-requests and the metadata processing scope corresponding to the sub-requests can be determined based on the ratio of the number of metadata index shards n obtained to the number of target node devices x plus one.

[0061] Step 406: Allocate each sub-request and the corresponding metadata processing scope to the corresponding target node devices so that each target node device can process the corresponding metadata.

[0062] Optionally, the master node can send sub-requests to various target node devices, including itself, through network communication protocols (such as HTTP, TCP, etc.). Each sub-request includes the corresponding metadata processing scope and the filtering conditions for the lookup request, such as the prefix and time range mentioned above.

[0063] Step 408: Summarize the processing results of each node device to obtain the search result corresponding to the search request.

[0064] The search results are object data that meet the search request criteria, obtained after processing and summarizing the metadata. Optionally, the search results can be all video clips from camera 123 in Area A between 9:00 AM and 11:00 AM on October 1, 2024, which will ultimately be returned to the client in the form of an ordered list (e.g., up to 1000 objects) for the user to view.

[0065] Optionally, after all node devices have returned the processing results to the master node, the master node can perform operations such as merging and sorting on these processing results in the form of a list in its local memory to obtain the final search result corresponding to the search request, and return it to the client for the user to view.

[0066] The aforementioned metadata query method decomposes the search request into multiple sub-requests, allowing multiple target node devices to process the corresponding sub-requests in parallel. This significantly reduces processing time and improves the response efficiency of the search operation. Furthermore, it fully utilizes the computing resources of multiple node devices in the distributed system, making system resources more effectively used.

[0067] In one embodiment, such as Figure 5 As shown, based on the identification information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined, including:

[0068] Step 502: Determine the index sharding information of the metadata based on the identification information.

[0069] The metadata index sharding information details how the target bucket's metadata index is divided into shards, primarily including the number of shards and the identifier range of each shard. For example, if the target bucket "city-surveillance" has its metadata index divided into 100 shards, then "100 shards" represents the number of shards. Assuming there are 5 target node devices, node 1's processing shards are 0-19, node 2's processing shards are 20-39, and so on. "0-19" and "20-39" represent the corresponding identifier range information for each shard.

[0070] Optionally, upon receiving a lookup request, the master node will determine the specific index shard information by querying the system's metadata management module based on the identifier information of the target bucket's metadata. Since the metadata management module manages the index shard information of the target bucket's metadata, including the number range of each shard, the master node can obtain complete index shard information after sending a query request to the metadata management module.

[0071] Step 504: Based on the index sharding information and the number of target node devices, decompose the lookup request into multiple sub-requests and determine the metadata processing scope corresponding to each sub-request.

[0072] Optionally, the master node can first determine the number of shards that each target node device should process based on the ratio of the index shard information to the number of sub-requests. For example, assuming there are 100 shards and 5 sub-requests, each node should process 100 / 5 = 20 shards. Then, consecutive shard ranges are allocated to each target node device sequentially, thereby decomposing the lookup request into multiple sub-requests. Each sub-request contains the corresponding metadata processing range and the filtering conditions for the lookup request.

[0073] In this embodiment, the massive and complex search task is decomposed into multiple subtasks by using the index sharding information of the metadata and the number of target node devices, allowing multiple target node devices to process in parallel, thereby greatly shortening the processing time and improving the response efficiency of the search operation.

[0074] In one embodiment, such as Figure 6 As shown, the index sharding information includes the number of index shards and the index sharding range. Based on the index sharding information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing range corresponding to each sub-request is determined, including:

[0075] Step 602: Decompose the search request into multiple sub-requests based on the number of target node devices.

[0076] The number of sub-requests is determined based on the number of target node devices.

[0077] It is understandable that when the master node receives a lookup request, some of the nodes in the network layer may be in a working state and some may be in an idle state. In order to make full use of the computing resources of the idle nodes, the idle node devices in the system can be identified as target node devices. However, when processing the lookup request, the master node also needs to process it. Therefore, the number of sub-requests is the number of target node devices plus one.

[0078] Step 604: Determine the number of index units processed by each target node device based on the number of index shards and the number of target node devices.

[0079] Optionally, as mentioned above, the number of index units processed by each target node device can be determined based on the ratio of the number of index shards in the target bucket's metadata to the number of sub-requests. If the ratio is not even, it can be rounded up to ensure that each node device is allocated enough index shards to cover the entire search range, avoiding the situation where some index shards are not processed by any node device, thereby ensuring that the search request can completely process all relevant metadata.

[0080] Step 606: Determine the index sharding range corresponding to each sub-request according to the order of sub-requests, the index sharding range, and the number of unit indexes.

[0081] Optionally, the master node first sorts the sub-requests obtained from the decomposition; then, based on the previously determined number of unit indexes and the known index sharding range, it sequentially allocates the corresponding index sharding range to each sub-request.

[0082] For example, assuming the index shard range is from 0 to 99 and the unit index number is 20, then the index shard range corresponding to the first sub-request is 0 to 19, the index shard range corresponding to the second sub-request is 20 to 39, and so on; finally, the master node will send each sub-request and its corresponding index shard range information to the corresponding target node device through the network communication protocol.

[0083] In this embodiment, metadata processing tasks are reasonably allocated according to the number of target node devices and the index sharding situation, allowing multiple target node devices to process different index sharding ranges in parallel, which greatly shortens the response time of the lookup operation and makes full use of the parallel computing capabilities of the distributed system, thereby helping to improve the system's processing efficiency.

[0084] In one embodiment, the processing results of each node device are aggregated to obtain the search result corresponding to the search request, including: performing data filtering and sorting processing on the processing results of each node device to obtain the search result.

[0085] Optionally, after each target node device sends its processing results to the master control node, the master control node receives the processing results from all node devices, including its own, and organizes them into a data set. Then, the master control node filters the integrated data set according to the filtering conditions of the search request. For example, if the search request asks for products priced between 100 and 200 yuan, the master control node will traverse the above data set and remove records whose prices are not within this range. After completing the filtering, the master control node sorts the remaining data according to the sorting rules specified in the search request, such as sorting by sales volume from high to low or by transaction time, to obtain the final search results.

[0086] In this embodiment, by filtering data, data that does not meet the search request conditions in the processing results of each node device is removed, thereby improving the accuracy of the search results.

[0087] In one embodiment, such as Figure 7 As shown, the processing results of each node device are filtered and sorted to obtain the search results, including:

[0088] Step 702: Perform data filtering and sorting on the processing results of each node device to obtain the sorted processing results.

[0089] Optionally, based on the filtering conditions in the search request, the processing results returned by each node device can be checked one by one, and those that do not meet the filtering conditions can be removed to obtain the filtered processing results; then, sorting algorithms such as quicksort can be used to sort the filtered processing results to obtain the sorted processing results.

[0090] Step 704: Select a preset number of processing results from the sorted processing results as search results.

[0091] The preset quantity setting can be adjusted according to the user's actual needs to ensure that the final search results returned are the part that the user cares about most.

[0092] Optionally, a preset number of processing results can be selected sequentially, starting from the beginning of the sorted results, according to the sorted order. For example, if the preset number is 100, then the first 100 processing results in the sorted results will be selected as the final search results.

[0093] In this embodiment, data filtering ensures the accuracy of the final search results, while sorting the processed results and selecting a preset number of results ensures the convenience for users to view the search results when they are returned to the client.

[0094] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0095] Based on the same inventive concept, this application also provides a metadata query apparatus for implementing the metadata query method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more metadata query apparatus embodiments provided below can be found in the limitations of the metadata query method described above, and will not be repeated here.

[0096] In one embodiment, such as Figure 8 As shown, a metadata query device is provided, including: a search request receiving module 802, a search request decomposition module 804, a metadata processing module 806, and a search result determination module 808, wherein:

[0097] The lookup request receiving module 802 is used to determine the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system when a lookup request is received; the lookup request is a request to look up the target object data in the target bucket.

[0098] The lookup request decomposition module 804 is used to decompose the lookup request into multiple sub-requests based on the identification information and the number of target node devices, and to determine the metadata processing scope corresponding to each sub-request.

[0099] The metadata processing module 806 is used to allocate each sub-request and the metadata processing scope corresponding to each sub-request to the corresponding target node device, so that each target node device can process the corresponding metadata.

[0100] The search result determination module 808 is used to summarize the processing results of each node device and obtain the search result corresponding to the search request.

[0101] In one embodiment, the lookup request decomposition module 804 is further configured to: determine the index sharding information of metadata based on the identification information; decompose the lookup request into multiple sub-requests based on the index sharding information and the number of target node devices, and determine the metadata processing scope corresponding to each sub-request.

[0102] In one embodiment, the lookup request decomposition module 804 is further configured to: decompose the lookup request into multiple sub-requests based on the number of target node devices; the number of sub-requests is determined based on the number of target node devices; determine the number of unit indexes processed by each target node device based on the number of index shards and the number of target node devices; and determine the index shard range corresponding to each sub-request according to the order of the sub-requests, the index shard range, and the number of unit indexes.

[0103] In one embodiment, the apparatus is further configured to: identify an idle node device in the distributed system as a target node device.

[0104] In one embodiment, the search result determination module 808 is further configured to: perform data filtering and sorting on the processing results of each node device to obtain the search result.

[0105] In one embodiment, the search result determination module 808 is further configured to: perform data filtering and sorting on the processing results of each node device to obtain sorted processing results; and determine a preset number of processing results from the sorted processing results as search results.

[0106] Each module in the aforementioned metadata query device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0107] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores lookup request data, metadata for various buckets within the system, and other data related to metadata queries. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a metadata query method.

[0108] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0110] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0111] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A metadata query method, characterized in that, The method includes: Upon receiving a search request, the system determines the identifier information of the metadata of the target bucket corresponding to the search request and the number of target node devices in the distributed system; the search request is a request to search for target object data within the target bucket. Based on the identification information and the number of target node devices, the search request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined; Each sub-request and the corresponding metadata processing scope are allocated to the corresponding target node device, so that each target node device can process the corresponding metadata. The processing results of each node device are summarized to obtain the search result corresponding to the search request.

2. The method according to claim 1, characterized in that, The step of decomposing the search request into multiple sub-requests based on the identification information and the number of target node devices, and determining the metadata processing scope corresponding to each sub-request, includes: Based on the identification information, determine the index fragmentation information of the metadata; Based on the index sharding information and the number of target node devices, the lookup request is decomposed into multiple sub-requests, and the metadata processing scope corresponding to each sub-request is determined.

3. The method according to claim 2, characterized in that, The index sharding information includes the number of index shards and the index sharding range; the step of decomposing the lookup request into multiple sub-requests based on the index sharding information and the number of target node devices, and determining the metadata processing range corresponding to each sub-request, includes: The search request is broken down into multiple sub-requests based on the number of target node devices; the number of sub-requests is determined based on the number of target node devices. The number of index units processed by each target node device is determined based on the number of index shards and the number of target node devices. The index sharding range corresponding to each sub-request is determined according to the order of the sub-requests, the index sharding range, and the unit index number.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The idle node device in the distributed system is identified as the target node device.

5. The method according to any one of claims 1 to 3, characterized in that, The process of summarizing the processing results of each node device to obtain the search result corresponding to the search request includes: The processing results of each node device are filtered and sorted to obtain the search results.

6. The method according to claim 5, characterized in that, The process of filtering and sorting the processing results of each of the node devices to obtain the search result includes: The processing results of each node device are filtered and sorted to obtain sorted processing results. A preset number of processing results from the sorted processing results are determined as the search results.

7. A metadata query device, characterized in that, The device includes: The lookup request receiving module is used to determine, upon receiving a lookup request, the identifier information of the metadata of the target bucket corresponding to the lookup request and the number of target node devices in the distributed system; the lookup request is a request to look up target object data within the target bucket. The lookup request decomposition module is used to decompose the lookup request into multiple sub-requests based on the identification information and the number of target node devices, and to determine the metadata processing scope corresponding to each sub-request. The metadata processing module is used to allocate each sub-request and the metadata processing scope corresponding to each sub-request to the corresponding target node device, so that each target node device can process the corresponding metadata. The search result determination module is used to summarize the processing results of each node device to obtain the search result corresponding to the search request.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images