Metadata processing method and device based on distributed cluster, equipment and medium

Abstract

The invention discloses a metadata processing method and device based on a distributed cluster, equipment and a medium. The method is executed by a cluster main replica node and comprises the following steps: receiving a metadata operation request sent by a client; wherein the metadata operation request carries a request parameter; determining an execution plan instance based on the request parameter and an execution plan template library; wherein the execution plan instance comprises a plurality of slave replica nodes; performing metadata processing operation based on the execution plan instance and the plurality of slave replica nodes, and determining execution response information of the plurality of slave replica nodes; and determining a request processing result according to the execution response information of each slave copy node, and feeding back the request processing result to the client. According to the technical scheme, the metadata is processed according to the execution plan instance, so that strong consistency between data redundant copies is ensured, a data consensus process and log recovery do not need to be carried out after the distributed cluster is restarted, and the recovery time is effectively shortened.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for metadata processing based on a distributed cluster. Background Technology

[0002] With the development of cloud computing and big data, distributed storage systems are widely used. To ensure service continuity and user experience, the recovery time (RTO) after a distributed cluster restart is getting higher and higher. However, the distributed multi-replica or EC data redundancy technology introduced for data security has brought even greater challenges to the recovery time of distributed clusters.

[0003] Currently, data processing and recovery techniques are largely based on log systems for persistence. However, a drawback of log systems is that even after data is persisted, invalid data cannot be immediately removed, leading to unnecessary operations during recovery. In a distributed environment, the completeness of each node's logs becomes unknown after a system failure. This necessitates the introduction of distributed consensus protocols. Before data recovery, distributed systems need to use consensus protocols (such as Paxos or Raft) to ensure that the logs used for recovery are up-to-date and complete. This undoubtedly exacerbates the time-to-operation (RTO) requirement for recovery. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for metadata processing based on a distributed cluster. By processing metadata according to execution plan instances, it ensures strong consistency among redundant data replicas and eliminates the need for data consensus processes and log recovery after a distributed cluster restart, effectively shortening recovery time.

[0005] According to one aspect of the present invention, a metadata processing method based on a distributed cluster is provided, the method being executed by a cluster master / replica node, comprising:

[0006] Receive a metadata operation request sent by the client; wherein the metadata operation request carries request parameters;

[0007] An execution plan instance is determined based on the request parameters and the execution plan template library; wherein, the execution plan instance includes multiple replica nodes;

[0008] Based on the execution plan instance and the plurality of slave replica nodes, metadata processing operations are performed, and the execution response information of the plurality of slave replica nodes is determined;

[0009] The request processing result is determined based on the execution response information of each replica node and fed back to the client.

[0010] According to another aspect of the present invention, a metadata processing apparatus based on a distributed cluster is provided, the apparatus being configured on a primary replica node of the cluster, comprising:

[0011] The request receiving module is used to receive metadata operation requests sent by the client; wherein the metadata operation request carries request parameters;

[0012] An execution plan instance determination module is used to determine an execution plan instance based on the request parameters and the execution plan template library; wherein, the execution plan instance includes multiple replica nodes;

[0013] The response information determination module is used to perform metadata processing operations based on the execution plan instance and the multiple slave replica nodes, and to determine the execution response information of the multiple slave replica nodes;

[0014] The request processing result determination module is used to determine the request processing result based on the execution response information of each replica node and feed it back to the client.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the metadata processing method based on a distributed cluster as described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the metadata processing method based on a distributed cluster as described in any embodiment of the present invention.

[0020] The technical solution of this invention involves receiving a metadata operation request sent by a client; wherein the metadata operation request carries request parameters; determining an execution plan instance based on the request parameters and an execution plan template library; wherein the execution plan instance includes multiple slave replica nodes; performing metadata processing operations based on the execution plan instance and the multiple slave replica nodes, and determining the execution response information of the multiple slave replica nodes; determining the request processing result based on the execution response information of each slave replica node and feeding it back to the client. This technical solution, by processing metadata according to the execution plan instance, ensures strong consistency among redundant data replicas and also eliminates the need for data consensus processes and log recovery after a distributed cluster restart, effectively shortening recovery time.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a metadata processing method based on a distributed cluster according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a metadata processing method for a distributed cluster according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a metadata processing device based on a distributed cluster according to Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1 This is a flowchart of a metadata processing method based on a distributed cluster according to Embodiment 1 of the present invention. This embodiment is applicable to the processing of metadata in a distributed cluster. The method can be executed by a metadata processing device based on a distributed cluster. This metadata processing device based on a distributed cluster can be implemented in hardware and / or software, and can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0031] The technical solution in this embodiment is applied in a distributed storage environment and can be executed by the cluster master and replica nodes. At the data layer, each node in this example has a corresponding replica node. For data layer operations, the master and replica nodes can reside on the cluster master node or be placed on other cluster slave nodes (cluster backup nodes). From a data layer perspective, master nodes, slave nodes, and master and replica nodes are equivalent and do not distinguish between master and slave. Each node can be a master and replica node to process metadata, thus ensuring load balancing when dealing with large amounts of data. It is understandable that the metadata replica can reside on the master node, and the master and replica nodes can reside on other slave nodes, achieving load balancing. Deploying a metadata management service in the distributed storage system records the data replica information of each storage node. When any storage node fails, the persistence status of the node can be determined based on the current execution response, allowing for rapid data replication and shortening data recovery time.

[0032] In this embodiment, the cluster master / replica node can be considered as the node to which the system sends metadata operation requests based on load balancing, and that node can then act as the master / replica node. Any node can receive the corresponding metadata operation request, thus becoming the master / replica node, and then distribute it to other nodes that need to process it, which can then be considered slave / replica nodes.

[0033] S110: Receive metadata operation requests sent by the client.

[0034] The metadata operation request carries request parameters. A metadata operation request can refer to a request for specific processing operations on metadata. In this embodiment, the metadata operation request may include specific request parameter information. Request parameters may include parameters such as operation type, target object, permission verification, and data location. For example, the operation type included in the request parameters can be considered the specific processing behavior requested, which may be processing operations such as creating metadata, querying metadata, modifying metadata, and allocating data space. In this embodiment, the request for specific processing operations on metadata sent by the client can be obtained.

[0035] S120. Determine the execution plan instance based on the request parameters and the execution plan template library.

[0036] The execution plan instance includes multiple replica nodes. The execution plan template library can refer to a database containing multiple execution plan templates. Each execution plan template encapsulates the merged execution steps of metadata operations. In this embodiment, the execution plan template library can be pre-generated. In this embodiment, a distributed storage server can generate various execution plan templates based on a predetermined metadata operation strategy, thereby constructing the execution plan template library. An execution plan instance can be considered a specific sequence of operation steps. In this embodiment, the execution plan instance can include a specific sequence of operation steps and multiple replica nodes involved in each operation step. In this embodiment, the number of replica nodes can be determined according to the specific operation steps. In this embodiment, the execution plan instance can be a template presented as an executable instance. In this embodiment, a specific sequence of operation steps for the current request can be dynamically generated based on the client's request parameters and predefined execution plan templates.

[0037] In this embodiment, optionally, determining an execution plan instance based on request parameters and an execution plan template library includes: determining a corresponding execution plan template from the execution plan template library based on request parameters; filling the request parameters into the execution plan template to generate an execution plan instance.

[0038] The execution plan template can be considered as the execution plan template determined based on the request parameters. It is understood that in this embodiment, different execution plan templates can be determined based on different request parameters.

[0039] In this embodiment, after receiving a client operation request, the cluster's primary and replica nodes input the request parameters into the corresponding execution plan template to generate an execution plan instance. Specifically, in this embodiment, the execution plan template corresponding to the request parameters can be determined from the execution plan template library based on the request parameters carried in the metadata operation request sent by the client. Then, the request parameters are filled into the corresponding execution plan template to generate an executable plan instance. The execution plan instance generation process in this embodiment includes template matching, parameter binding, dependency resolution, and resource allocation operations.

[0040] In this embodiment, the abstract operation can be transformed into a specific executable step by such a setting, which ensures that the system can execute according to the expected logic, while supporting flexible expansion and fault tolerance, thereby enabling efficient and reliable processing of complex metadata operation requests.

[0041] S130. Perform metadata processing operations based on the execution plan instance and multiple slave replica nodes, and determine the execution response information of multiple slave replica nodes.

[0042] Metadata processing operations refer to the operations performed by different replica nodes on metadata according to the execution plan instance. Execution response information can be understood as the response information from each replica node indicating whether the metadata processing operation was successfully executed. In this embodiment, the execution response information may include success response information. It can be understood that success response information refers to the response from the replica node indicating that the operation on metadata was successfully executed according to the plan. In this embodiment, if the replica node fails to successfully execute the operation on metadata according to the plan, it may not provide response information or may provide a response indicating unsuccessful execution; this can be configured according to actual needs.

[0043] In this embodiment, multiple replica nodes can be controlled to perform operations according to the corresponding steps in the execution plan instance based on the specific operation steps in the execution plan instance and the multiple replica nodes involved. After successful execution, they can send a success response message to the primary replica node, thereby determining the execution response information of multiple replica nodes.

[0044] S140. Determine the request processing result based on the execution response information of each replica node and send it back to the client.

[0045] The request processing result can be understood as the request response result returned to the client. In this embodiment, the request processing result may refer to whether the execution response information returned by each replica node meets the set threshold conditions.

[0046] In this embodiment, the primary replica node can determine the processing result based on the execution response information returned by each secondary replica node. This is achieved by comparing the number of successful execution responses with a set successful replica threshold. The result is then used to determine the corresponding request processing result and send it back to the client. For example, if the number of successful execution responses is greater than or equal to the set successful replica threshold, the request processing result is considered complete, and the client is notified of the success. Conversely, if the number of successful execution responses is less than the set successful replica threshold, the request processing result is considered incomplete, and the client is notified of the failure.

[0047] The technical solution of this invention involves receiving a metadata operation request sent by a client; wherein the metadata operation request carries request parameters; determining an execution plan instance based on the request parameters and an execution plan template library; wherein the execution plan instance includes multiple slave replica nodes; performing metadata processing operations based on the execution plan instance and the multiple slave replica nodes, and determining the execution response information of the multiple slave replica nodes; determining the request processing result based on the execution response information of each slave replica node and feeding it back to the client. This technical solution, by processing metadata according to the execution plan instance, ensures strong consistency among redundant data replicas and also eliminates the need for data consensus processes and log recovery after a distributed cluster restart, effectively shortening recovery time.

[0048] Example 2

[0049] Figure 2 This is a flowchart of a metadata processing method in a distributed cluster according to Embodiment 2 of the present invention. This embodiment is based on the above embodiment but with optimizations. Specifically, the optimization includes: before receiving the metadata operation request sent by the client, obtaining multiple operation request parameters of the metadata; determining multiple corresponding execution plan templates based on the multiple operation request parameters; and assembling the multiple execution plan templates into an execution plan template library. Figure 2 As shown, the method includes:

[0050] S210, Multiple operation request parameters for obtaining metadata.

[0051] The multiple operation request parameters can refer to specific operation parameters for metadata operations. In this embodiment, the multiple operation request parameters can be some basic operation request parameters for metadata, which may include parameter information such as operation type, target resource, user permissions, and resource attributes. Specifically, in this embodiment, the operation request parameters may include operation request parameters such as create, read, update, and delete. For example, specific operation request parameters may include operation request parameters such as creating a file / directory, reading file metadata or content, modifying file attributes (such as permissions or size), deleting a file or directory, and moving or renaming a file.

[0052] In this embodiment, some basic and conventional operation request parameters for processing metadata can be obtained in advance.

[0053] S220. Determine multiple execution plan templates based on multiple operation request parameters, and combine the multiple execution plan templates to obtain an execution plan template library.

[0054] Here, the execution plan template refers to the execution plan template generated based on specific operation request parameters. In this embodiment, execution plan templates can be generated according to the specific basic operation request parameters contained in the basic metadata operation strategy, thereby obtaining multiple execution plan templates, and assembling multiple execution plan templates into an execution plan template library.

[0055] For example, in this embodiment, for the metadata of the corresponding storage layer, the operation of allocating a space for the metadata is predictable. It is possible to predict what operations it will perform. Then, these operations are rehearsed on each node that needs to be processed. After the rehearsal, a template is formed. It is equivalent to determining the corresponding execution plan template based on the corresponding basic operation request parameters. Thus, multiple corresponding execution plan templates can be generated in advance through rehearsal based on different operation request parameters.

[0056] S230: Receive metadata operation requests sent by the client.

[0057] Metadata operation requests carry request parameters.

[0058] S240. Determine the execution plan instance based on the request parameters and the execution plan template library.

[0059] The execution plan instance includes multiple replica nodes.

[0060] S250: Perform metadata processing operations based on the execution plan instance and multiple slave replica nodes, and determine the execution response information of multiple slave replica nodes.

[0061] In this embodiment, optionally, metadata processing operations are performed based on the execution plan instance and multiple slave replica nodes, and the execution response information of multiple slave replica nodes is determined, including: distributing the execution plan instance to each slave replica node in the cluster, so that each slave replica node performs data processing operations on the metadata according to the execution plan instance and feeds back execution response information.

[0062] In this context, data processing operations can be understood as the execution of corresponding metadata processing operations by replica nodes according to a planned execution instance. In this embodiment, the primary replica node can distribute the execution plan instance to each involved secondary replica node, enabling each secondary replica node to perform corresponding data processing operations on the metadata according to the execution plan and then send execution response information back to the primary replica node. It is understandable that if the data processing execution operation is successful, i.e., after persistence, a success response information can be returned to the primary replica node. For example, after each secondary replica node completes its execution, it will send back corresponding response information. For instance, if there are four secondary replica nodes, after the four secondary replica nodes perform data processing operations on the metadata according to the execution plan instance and obtain four execution response information, the plan can be considered to have been executed successfully.

[0063] As is understandable, in this embodiment, the execution of each replica node has already been written to the corresponding disk and has been persisted.

[0064] In this embodiment, the configuration allows each replica node to perform corresponding execution operations according to a pre-defined plan template. This enables each replica node to synchronously receive execution response information during execution, thus providing a clear understanding of the completion status of metadata processing.

[0065] S260. Determine the request processing result based on the execution response information of each replica node and send it back to the client.

[0066] In this embodiment, optionally, determining the request processing result and feeding it back to the client based on the execution response information of each replica node includes: determining the number of successful responses in the execution response information of each replica node; determining the request processing result based on the number of successful responses and a number threshold and feeding it back to the client.

[0067] The number of successful responses can be understood as the number of successful response messages contained in the execution response information. The number threshold can be a pre-set threshold for the number of successful responses. In this embodiment, the number threshold can be determined based on the number of slave replica nodes involved. For example, if the number of slave replica nodes is 4, the data threshold can be set to 3 or 2, which can be set according to the actual processing requirements.

[0068] In this embodiment, the primary replica node can count the number of successful response messages received from the execution response information of each secondary replica node, then compare the number of successful response messages with a pre-set number, determine the corresponding request processing result based on the comparison result, and feed the request processing result back to the client.

[0069] In this embodiment, the execution status of each replica node can be determined based on its response, thus ensuring consistency among redundant data replicas and facilitating timely determination of the execution status of each node.

[0070] In this embodiment, optionally, determining the request processing result and feeding it back to the client based on the number of successful responses and a quantity threshold includes: determining whether the number of successful responses is greater than or equal to the quantity threshold; if the number of successful responses is greater than or equal to the quantity threshold, then determining the request processing result as a successful response and feeding it back to the client.

[0071] In this embodiment, the quantity threshold can be the minimum number of slave replicas required to complete the task. In this embodiment, it can be determined whether the number of successful responses is greater than or equal to the pre-set minimum number of slave replicas to be completed. If the number of successful responses is greater than or equal to the minimum number of slave replicas, the request processing result is determined to be a successful response, and a successful response message is sent back to the client. If the number of successful responses is less than the minimum number of slave replicas, the request processing result is determined to be a failed response, and a failed response message is sent back to the client.

[0072] In this embodiment, the success of a client operation request can be determined based on the number of successful responses reported by each replica node and a pre-set threshold number, which improves the efficiency of operation request completion and makes it easier to understand the specific completion status of each node.

[0073] In this embodiment, optionally, it also includes: if an abnormal cluster restart operation is detected, loading the latest execution plan instance; determining whether the latest execution plan instance has been executed; if the latest execution plan instance has not been executed, rolling back multiple current execution plan instances from the replica nodes and re-executing the latest execution plan instance.

[0074] Cluster abnormal restart operation can be considered as an operation that requires restarting the cluster due to an abnormal situation. The latest execution plan instance can be considered as the most recent execution plan instance.

[0075] In this embodiment, if a cluster restart operation is detected after an anomaly, the latest execution plan instance can be loaded through the primary replica node. It is then determined whether the latest execution plan instance has completed execution. If the latest execution plan instance has not completed execution, it indicates that the metadata has not been persisted. In this case, multiple replica nodes in the cluster will roll back their current execution plan instances and restart the execution of the latest execution plan instance. If the latest execution plan instance has completed execution, it indicates that the metadata has been persisted, and each node can continue with the execution plan instance corresponding to the next operation request.

[0076] It is understandable that there are many execution plan instances in this embodiment, which are executed sequentially, and the execution plan instances are idempotent, so the result of repeated execution is the same.

[0077] In this embodiment, by setting it up in the event of an abnormal cluster restart, it is possible to determine whether the execution plan instance has been executed and whether the metadata has been persisted, thereby further ensuring the persistence and reliability of the data, ensuring the simplicity of data recovery, and ensuring the consistency of data between nodes, and shortening the data recovery time.

[0078] The technical solution of this invention involves: acquiring multiple operation request parameters of metadata; determining multiple corresponding execution plan templates based on the multiple operation request parameters, and assembling the multiple execution plan templates into an execution plan template library; receiving metadata operation requests sent by a client; wherein the metadata operation requests carry request parameters; determining execution plan instances based on the request parameters and the execution plan template library; wherein the execution plan instance includes multiple slave replica nodes; performing metadata processing operations based on the execution plan instance and the multiple slave replica nodes, and determining the execution response information of the multiple slave replica nodes; determining the request processing result based on the execution response information of each slave replica node and feeding it back to the client. This technical solution, by processing metadata according to execution plan instances, avoids the inherent defects of using log systems, ensures strong consistency among redundant data replicas, and effectively shortens recovery time by eliminating the need for data consensus processes and log recovery after a distributed cluster restart.

[0079] Example 3

[0080] Figure 3 This is a schematic diagram of a metadata processing device based on a distributed cluster according to Embodiment 3 of the present invention. The device is configured on the primary replica node of the cluster, such as... Figure 3 As shown, the device includes:

[0081] The request receiving module 310 is used to receive metadata operation requests sent by the client; wherein the metadata operation request carries request parameters;

[0082] The execution plan instance determination module 320 is used to determine the execution plan instance based on the request parameters and the execution plan template library; wherein, the execution plan instance includes multiple replica nodes;

[0083] The response information determination module 330 is used to perform metadata processing operations based on the execution plan instance and multiple slave replica nodes, and to determine the execution response information of multiple slave replica nodes;

[0084] The request processing result determination module 340 is used to determine the request processing result based on the execution response information of each replica node and feed it back to the client.

[0085] Optionally, the device further includes: an execution plan template library assembly module, used to obtain multiple operation request parameters of metadata before receiving metadata operation requests sent by the client; determine multiple corresponding execution plan templates based on the multiple operation request parameters; and assemble the multiple execution plan templates into an execution plan template library.

[0086] Optionally, the execution plan instance determination module 320 is specifically used for:

[0087] The corresponding execution plan template is determined from the execution plan template library based on the request parameters.

[0088] Fill the request parameters into the execution plan template to generate an execution plan instance.

[0089] Optionally, the response information determination module 330 is specifically used for:

[0090] The execution plan instance is distributed to each slave replica node in the cluster, so that each slave replica node performs data processing operations on the metadata according to the execution plan instance and sends back execution response information.

[0091] Optionally, the request processing result determination module 340 includes:

[0092] The quantity determination unit is used to determine the number of successful responses in the execution response information of each replica node;

[0093] The result determination unit is used to determine the request processing result based on the number of successful responses and a quantity threshold, and then feed it back to the client.

[0094] Optional, the result determination unit, specifically used for:

[0095] Determine if the number of successful responses is greater than or equal to the quantity threshold;

[0096] If the number of successful responses is greater than or equal to the threshold, the request processing result is determined to be a successful response and is sent back to the client.

[0097] Optionally, the device further includes: a restart processing module, used to load the latest execution plan instance if an abnormal cluster restart operation is detected; determine whether the latest execution plan instance has been executed; and if the latest execution plan instance has not been executed, roll back multiple current execution plan instances from the replica nodes and re-execute the latest execution plan instance.

[0098] The metadata processing device based on a distributed cluster provided in this embodiment of the invention can execute a metadata processing method based on a distributed cluster provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0099] Example 4

[0100] Figure 4 This is a schematic diagram of an electronic device according to Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0101] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as metadata processing methods based on distributed clusters.

[0104] In some embodiments, the distributed cluster-based metadata processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the distributed cluster-based metadata processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the distributed cluster-based metadata processing method by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A metadata processing method based on a distributed cluster, characterized in that, The method is executed by the cluster's primary replica node and includes: Receive a metadata operation request sent by the client; wherein the metadata operation request carries request parameters; An execution plan instance is determined based on the request parameters and the execution plan template library; wherein, the execution plan instance includes multiple replica nodes; Based on the execution plan instance and the plurality of slave replica nodes, metadata processing operations are performed, and the execution response information of the plurality of slave replica nodes is determined; The request processing result is determined based on the execution response information of each replica node and fed back to the client.

2. The method according to claim 1, characterized in that, Before receiving metadata operation requests from clients, the following is also included: Multiple operation request parameters to retrieve metadata; Based on the multiple operation request parameters, determine the corresponding multiple execution plan templates, and assemble the multiple execution plan templates to obtain an execution plan template library.

3. The method according to claim 1, characterized in that, Determining an execution plan instance based on the request parameters and the execution plan template library includes: Based on the request parameters, a corresponding execution plan template is determined from the execution plan template library; The request parameters are filled into the execution plan template to generate an execution plan instance.

4. The method according to claim 1, characterized in that, Based on the execution plan instance and the plurality of slave replica nodes, metadata processing operations are performed, and the execution response information of the plurality of slave replica nodes is determined, including: The execution plan instance is distributed to each slave replica node in the cluster, so that each slave replica node performs data processing operations on the metadata according to the execution plan instance and feeds back execution response information.

5. The method according to claim 1, characterized in that, The request processing result is determined based on the execution response information of each replica node and fed back to the client, including: Determine the number of successful responses in the execution response information of each replica node; The request processing result is determined based on the number of successful responses and the number threshold, and then fed back to the client.

6. The method according to claim 5, characterized in that, The request processing result is determined based on the number of successful responses and the number threshold, and then fed back to the client, including: Determine whether the number of successful responses is greater than or equal to a quantity threshold; If the number of successful responses is greater than or equal to the threshold, the request processing result is determined to be a successful response and is fed back to the client.

7. The method according to claim 1, characterized in that, Also includes: If an abnormal cluster restart is detected, the latest execution plan instance is loaded. Determine whether the latest execution plan instance has completed execution; If the latest execution plan instance is not completed, multiple replica nodes will roll back the current execution plan instance and re-execute the latest execution plan instance.

8. A metadata processing device based on a distributed cluster, characterized in that, The device is configured on the cluster primary replica node and includes: The request receiving module is used to receive metadata operation requests sent by the client; wherein the metadata operation request carries request parameters; An execution plan instance determination module is used to determine an execution plan instance based on the request parameters and the execution plan template library; wherein, the execution plan instance includes multiple replica nodes; The response information determination module is used to perform metadata processing operations based on the execution plan instance and the multiple slave replica nodes, and to determine the execution response information of the multiple slave replica nodes; The request processing result determination module is used to determine the request processing result based on the execution response information of each replica node and feed it back to the client.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the metadata processing method based on a distributed cluster as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the metadata processing method based on any one of claims 1-7.

Images