A method for efficiently accessing mass unstructured metadata

Abstract

The application discloses a kind of high-efficiency access mass unstructured metadata method, comprising: step 1, IDC Client is initiated by IDC API to IDC Server and requests index metadata;Step 2, IDC Core is encapsulated into message by request and is sent to metadata index server by IDC Net, while initializing IDC Async, prepares asynchronous response to obtain index metadata;Step 3, IDC Server receives IDC API request, generates asynchronous query flag and returns to IDC Client immediately, while asynchronously building query processing, and result set is stored in server IDC Cache;Step 4, IDC Core is requested index metadata by asynchronous round-robin to IDC Server by query flag, and is stored in IDC Cache;Step 5, IDC Client is inquired from IDC Cache and the index metadata required by business.The application introduces the cache basic component of custom in client and server, greatly shortens the network IO of main time-consuming on metadata request link;The performance of obtaining mass unstructured metadata is improved, and the overall availability and concurrency of upper data application business are promoted.

Description

Technical Field

[0001] This invention relates to the field of data backup technology, and more specifically to a method for efficiently accessing massive amounts of unstructured metadata. Background Technology

[0002] In the field of data backup technology, the backup and recovery of massive amounts of unstructured data typically requires high-performance metadata system services for efficient mapping and indexing to meet the needs of rapid incremental backups and fine-grained rapid recovery, thereby enabling fast retrieval, querying, and insertion operations. However, in scenarios with massive amounts of data, discrete network request I / O can severely impact the overall system throughput, thereby reducing the speed of data backup and recovery. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a method for efficiently accessing massive amounts of unstructured metadata. By employing a custom caching architecture model, it efficiently merges discrete network request I / O, thereby improving data backup performance and effectively supporting massive data business scenarios.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solution: This invention provides a method for efficiently accessing massive amounts of unstructured metadata, the specific steps of which are as follows: Step 1: The IDC client initiates an index metadata request to the IDC server through the IDC application programming interface; Step 2: The IDC core module encapsulates the request from Step 1 into a message and sends it to the metadata index server through the IDC network communication module. At the same time, it initializes the IDC asynchronous module to prepare for asynchronous response and obtain index metadata. Step 3: The IDC server receives the IDC application interface request, generates an asynchronous query flag and immediately returns it to the IDC client. At the same time, it asynchronously constructs the query processing and stores the result set in the server's IDC cache module. Step 4: The IDC core module asynchronously and pollingly requests index metadata from the IDC server using the query flag returned in Step 3, and stores it in the local IDC cache module. Step 5: The IDC client queries the index metadata required by the business from the local IDC cache module.

[0005] A further improvement of the present invention is that the IDC client and the IDC server use an asynchronous double-ended blocking queue mechanism.

[0006] A further improvement of the present invention is that the index metadata request in step 1 is a control flow message carrying a cache name parameter, which is used to identify an independent cache queue on the IDC server.

[0007] A further improvement of the present invention is that: after receiving the index metadata request in step 3, the IDC server randomly generates a globally unique identifier ID based on the received current cache name parameter, and combines it with the cache name to form a unique identifier under the current request session. This unique identifier is returned to the IDC client in a control flow manner. At the same time, the server asynchronously creates a corresponding cache producer worker thread based on the unique identifier. This worker thread is responsible for asynchronously retrieving metadata information from the IDC server database, organizing this metadata into a metadata object, and adding it to the blocking queue named by the unique identifier.

[0008] The beneficial effects of the present invention are: (1) Through the asynchronous double-ended blocking queue mechanism, flow control can be implemented on the one hand; on the other hand, the overall throughput of the system can be improved (the internal core uses a kernel-level synchronization mechanism, which reduces the switching cost between kernel mode and user mode in high-concurrency scenarios). (2) By designing efficient and flexible communication, caching, and query data structures, the system can be made more customizable, more robust, and easier to expand. (3) The multi-link concurrent model based on IO multiplexing architecture better supports the server's cache query capability. Attached Figure Description

[0009] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0010] Figure 2 This is an architecture diagram of an embodiment of the method of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0012] like Figure 1-2As shown, this invention provides a method for efficiently accessing massive amounts of unstructured metadata. It fully utilizes existing I / O multiplexing technology and, based on this, designs a unique communication request identifier. On the IDC Server side, it combines a multi-level feedback queue to construct a fast-response request model and a high-concurrency architecture model for intensive requests, maximizing the IDC Server's cache processing capability. This invention includes an Index Data Management (hereinafter referred to as IDC) API (IDC Application Programming Interface), IDC Client, IDC Core, IDC Net, IDC Async, IDC Server, and IDC Cache. The IDC Core is a core working module within the IDC Client; it does not interact with external systems but only with the IDC server internally, eliminating the need for direct interaction between the IDC client and the IDC server.

[0013] The specific steps of this invention are as follows: Step 1: The IDC Client sends an index metadata request to the IDC Server via the IDC API; Step 2: IDC Core encapsulates the index metadata request from Step 1 into a message and sends it to the metadata index server via IDC Net. At the same time, it initializes IDC Async to prepare for asynchronous response to obtain index metadata. Step 3: The IDC Server receives the IDC API request, generates an asynchronous query flag and immediately returns it to the IDCClient. At the same time, it asynchronously constructs the query processing and stores the result set in the server-side IDC Cache. Step 4: IDC Core asynchronously polls the IDC Server for index metadata using the query flag returned in Step 3 and stores it in the local IDC Cache; Step 5: The IDC Client queries the index metadata required by the business from the local IDC Cache.

[0014] In this invention, the IDC Client and IDC Server use an asynchronous double-ended blocking queue mechanism to fully realize efficient asynchronous concurrency and improve the overall system throughput.

[0015] like Figure 2As shown, the IDC client sends a metadata information request to the IDC server through the IDC application programming interface. This request does not involve the actual transmission of metadata entity information, but is only a control flow message. The control message carries a cache name parameter, which is used to identify an independent cache queue on the server side, so as to achieve high-performance response on the server side under high concurrency requests. After receiving the metadata control flow message, the IDC server randomly generates a globally unique identifier (ID) based on the received current cache name parameter. This ID is then combined with the cache name to form a unique identifier for the current request session, which is returned to the IDC client via control flow. Simultaneously, a corresponding cache producer worker thread is asynchronously created based on the unique identifier. This worker thread is responsible for asynchronously retrieving metadata information from the IDC server's database, organizing this metadata into metadata objects, and adding them to a blocking queue named after the unique identifier (one queue corresponds to one unique identifier, and the unique identifier is used as the key in a HashMap for unified management and storage). This allows the IDC client to asynchronously perform multiple queries for the same unique identifier within the same session. Corresponding to the producer worker thread, there is also a consumer worker thread responsible for asynchronously consuming metadata objects from the blocking queue and listening for metadata data stream requests sent by the IDC client. The IDC client will first receive a response to the control flow request from the IDC server, which includes a unique identifier (globally unique identifier ID + cache name). The IDC client then sends a metadata data stream request to the IDC server using a unique identifier; When the IDC server receives a metadata data stream request from the IDC client, it uses a unique identifier parameter and the consumer thread to query the corresponding blocking queue in the Hash Map, retrieves the metadata object in the blocking queue, and returns it to the client. After receiving the data stream response from the IDC server, the IDC client parses the metadata object and packages it into a Hash Map (with the metadata name of the specific request as the key and value) for asynchronous querying by the client; At this point, the asynchronous metadata request and response between the IDC client and the IDC server is complete. It can be seen that the entire query operation is performed asynchronously, and the client will not be interrupted from other services by the request. The server can also handle requests from multiple clients. Overall, the throughput and concurrency have good robustness.

[0016] The above description is merely a specific embodiment of the present invention, and further details the purpose, technical solution, and beneficial effects of the present invention. Finally, it should be noted that the above description is only a preferred embodiment of the present invention and does not impose any limitations on the present invention. For those skilled in the art, any non-innovative modifications, variations, and alterations made to the technical solution of the present invention based on the above content without departing from the scope of the present invention should also be considered within the protection scope of the present invention.

Claims

1. A method for efficiently accessing massive amounts of unstructured metadata, characterized in that: Includes the following steps: Step 1: The IDC client initiates an index metadata request to the IDC server through the IDC application programming interface. This request does not involve the actual transmission of metadata entity information, but is only a control flow message. This control flow message carries a cache name parameter, which is used to identify an independent cache queue on the server. Here, IDC is responsible for index data management. Step 2: The IDC core module encapsulates the index metadata request from Step 1 into a message and sends it to the metadata index server on the IDC server through the IDC network communication module. At the same time, it initializes the IDC asynchronous module on the IDC server to prepare for asynchronous response to obtain index metadata. Step 3: The IDC server receives the IDC application interface request, generates an asynchronous query flag and immediately returns it to the IDC client. At the same time, it asynchronously constructs the query processing and stores the result set in the server's IDC cache module. Step 4: The IDC core module of the IDC client asynchronously and pollingly requests index metadata from the IDC server using the asynchronous query flag returned in Step 3, and stores it in the local IDC cache module. Step 5: The IDC client queries the index metadata required by the business from the local IDC cache module.

2. The method for efficiently accessing massive amounts of unstructured metadata according to claim 1, characterized in that: The IDC client and IDC server use an asynchronous double-ended blocking queue mechanism.

3. The method for efficiently accessing massive amounts of unstructured metadata according to claim 1, characterized in that: In step 3, after receiving the index metadata request, the IDC server randomly generates a globally unique identifier ID based on the received current cache name parameter. This ID is then combined with the cache name to form a unique identifier for the current request session, i.e., an asynchronous query identifier. This identifier is returned to the IDC client in a control flow manner. Simultaneously, a corresponding cache producer worker thread is asynchronously created based on the unique identifier. This worker thread is responsible for asynchronously retrieving metadata information from the IDC server database, organizing this metadata into a metadata object, and adding it to the blocking queue named by the unique identifier.

Images