Big data cluster-oriented management method, device, system, equipment and medium

Abstract

The invention provides a management method, device, system and equipment for a big data cluster and a medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of intelligent cloud, big data, cloud computing and the like. The big data cluster-oriented management method is applied to a unified platform, the unified platform is connected with a plurality of candidate big data clusters, and the method comprises the following steps: intercepting a management request for a target big data cluster in the plurality of candidate big data clusters; the management request comprises identification information of the target big data cluster; obtaining metadata of the target big data cluster based on the identification information; and based on the metadata, managing the target big data cluster.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the technical fields of intelligent cloud, big data, cloud computing, etc., and in particular to a management method, device, system, equipment, medium and product for big data clusters. Background Art

[0002] In big data scenarios, faced with complex and changing business needs, enterprises usually need different types of big data clusters.

[0003] How to manage multiple big data clusters is a problem that needs to be solved. Summary of the invention

[0004] The present disclosure provides a management method, apparatus, system, device, medium and product for a big data cluster.

[0005] According to one aspect of the present disclosure, a management method for big data clusters is provided, which is applied to a unified platform, wherein the unified platform is connected to multiple candidate big data clusters, and the method comprises: intercepting a management request for a target big data cluster among the multiple candidate big data clusters; the management request includes identification information of the target big data cluster; based on the identification information, obtaining metadata of the target big data cluster; and managing the target big data cluster based on the metadata.

[0006] According to another aspect of the present disclosure, a management device for a big data cluster is provided, which is applied to a unified platform, wherein the unified platform is connected to multiple candidate big data clusters, and the device includes: an interception module, which is used to intercept management requests for a target big data cluster among the multiple candidate big data clusters; the management request contains identification information of the target big data cluster; an acquisition module, which is used to acquire metadata of the target big data cluster based on the identification information; and a management module, which is used to manage the target big data cluster based on the metadata.

[0007] According to another aspect of the present disclosure, a big data system is provided, including: a plurality of candidate big data clusters; a unified platform connected to the plurality of candidate big data clusters; the unified platform is as described in any one of the above items.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.

[0011] The present disclosure can improve the management effect of big data clusters.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of an implementation architecture for implementing an embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram of data interaction provided according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a second embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0020] Figure 7 It is a schematic diagram of an electronic device used to implement the management method for big data clusters according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Big data cluster is a powerful and flexible basic platform for storing, processing and analyzing massive data.

[0023] Big data clusters can include the following types:

[0024] Big data clusters based on Hadoop (referred to as Hadoop clusters): This type of cluster integrates key components such as HDFS (Hadoop Distributed File System), Yarn (Yet Another Resource Negotiator), MapReduce (a programming model for parallel computing of large data sets (greater than 1TB)), Zookeeper (a distributed coordination service), Hive (a data warehouse tool based on Hadoop that provides data summarization, query and analysis), and Spark (a fast and general large-scale data processing engine), forming a complete solution that integrates data storage, resource management, task scheduling, and data analysis. This cluster architecture can not only meet the needs of large-scale data storage, but also significantly improve the speed and efficiency of data processing through parallel computing.

[0025] Kafka-based big data cluster (referred to as Kafka cluster): Big data clusters built on Kafka have become the preferred platform for real-time data stream processing due to their high throughput, low latency, and high scalability. Such clusters usually include components such as Kafka (a distributed stream processing platform) and Zookeeper (for Kafka cluster management and configuration), which can easily cope with the real-time collection, transmission, and processing needs of massive data.

[0026] Big data cluster based on Doris (referred to as Doris cluster): Doris, as a high-performance MPP (Massively Parallel Processing) analytical database, has gradually emerged in the field of big data. The big data cluster built on Doris has become a powerful assistant in the field of data analysis with its excellent query performance, flexible expansion capabilities and rich data models. This type of cluster usually includes components such as Doris and Zookeeper, which can provide users with efficient and convenient data query and analysis services.

[0027] Ambari: An open source cluster management tool.

[0028] Although Ambari performs well in some aspects, its limitations become apparent when an enterprise has multiple independent clusters that are distributed in different physical locations or cloud environments. Since Ambari cannot directly support unified management across clusters, enterprises have to adopt a compromise solution: deploy multiple Ambari services, each responsible for managing a cluster.

[0029] In related technologies, when faced with multiple types of big data clusters, a decentralized management method is usually adopted. Different types of clusters need to log in to different management and control platforms for management, which not only increases the complexity and difficulty of management, but also greatly reduces the efficiency and effectiveness of management.

[0030] In order to improve the management effect of big data clusters, the present disclosure provides the following embodiments.

[0031] Figure 1 This embodiment provides a management method for a big data cluster, which is applied to a unified platform, and the unified platform is connected to multiple candidate big data clusters. The method includes:

[0032] 101. Intercept a management request for a target big data cluster among the multiple candidate big data clusters; the management request includes identification information of the target big data cluster.

[0033] 102. Based on the identification information, obtain metadata of the target big data cluster.

[0034] 103. Manage the target big data cluster based on the metadata.

[0035] In complex scenarios, multiple big data clusters of different types may be configured, and these big data clusters may be referred to as candidate big data clusters.

[0036] For example, multiple candidate big data clusters include: Hadoop cluster, Kafka cluster, and Doris cluster.

[0037] Since there are certain problems in the decentralized management of each big data cluster, in the embodiments of the present disclosure, a unified (or centralized) management of multiple big data clusters is performed based on a unified platform.

[0038] Based on this, a unified platform can be set up, and multiple candidate big data clusters are connected to the unified platform, that is, the management request of each candidate big data cluster is intercepted by the unified platform, and the unified platform manages multiple candidate big data clusters in a unified manner based on the intercepted management requests.

[0039] To this end, the management method can be executed by the unified platform.

[0040] The target big data cluster refers to a big data cluster currently to be managed among multiple candidate big data clusters.

[0041] A management request is used to manage the target big data cluster. The management request is sent by a user through a client, for example.

[0042] Management requests can be used to change the relevant configuration information of the target big data cluster. For example, if the target big data cluster is a Hadoop cluster, the configuration parameters of Hadoop components can be modified, such as adjusting the block size of HDFS (Hadoop Distributed FileSystem) and the resource allocation strategy of Yarn (Yet Another Resource Negotiator). These configuration changes can be updated synchronously in the entire cluster to ensure the configuration consistency of all nodes. This eliminates the need to operate each node in the cluster one by one, thereby improving management efficiency.

[0043] Identification information (ID) is used to uniquely identify a big data cluster. Different types of big data clusters have different identification information.

[0044] For example, if the Hadoop cluster is to be managed currently, the management request carries the identification information of the Hadoop cluster.

[0045] The management request may specifically be a Hypertext Transfer Protocol (HTTP) request, based on which the identification information may be included in a Uniform Resource Locator (URL) parameter or a request body of the HTTP request.

[0046] After obtaining the identification information of the target big data cluster carried in the management request, metadata of the target big data cluster may be obtained based on the identification information.

[0047] Specifically, a correspondence between identification information and metadata may be preconfigured, and based on the correspondence, metadata corresponding to the identification information carried in the management request may be obtained.

[0048] Metadata is used to describe the big data cluster, including, for example, the cluster name, Internet Protocol (IP) address, port, security credentials, configuration parameters, etc.

[0049] After obtaining the metadata of the target big data cluster, the target big data cluster is managed based on the metadata. For example, based on the cluster name, the management request is forwarded to the corresponding target big data cluster, and the return data obtained by the target big data cluster based on the management request is received, and the return data is forwarded to the sender of the management request.

[0050] In this embodiment, multiple candidate big data clusters are managed based on a unified platform, which can achieve centralized, unified and efficient management, reduce management difficulty and cost, and thus improve management effect.

[0051] In order to better understand the embodiments of the present disclosure, the application scenarios of the embodiments of the present disclosure are described as follows.

[0052] Figure 2 It is a schematic diagram of an implementation architecture for implementing the embodiments of the present disclosure.

[0053] like Figure 2 As shown, the architecture includes: an application layer 201 , a platform layer 202 , and an engine layer 203 .

[0054] The application layer 201 is used to provide a user interface, such as providing a management interface for managing a big data cluster to users through a web page (web), a client, etc.

[0055] The platform layer 202 is used to uniformly manage multiple candidate big data clusters, and combined with the previous embodiment, it can be called a unified platform. The platform can specifically complete one or more of the following management: service management, log management, resource management, and configuration management.

[0056] The engine layer 203 is used to provide multiple candidate big data clusters, such as Hadoop cluster, Kafka cluster, Doris cluster, etc.

[0057] Management requests generated by the application layer, such as HTTP requests generated by users through web pages, are intercepted by the platform layer (unified platform). The platform layer obtains the metadata of the target big data cluster based on the identification information of the target big data cluster carried in the management request, and forwards the management request to the target big data cluster based on the metadata. The target big data cluster processes the management request and obtains return data, which is sent to the platform layer, which displays it to the user through the application layer.

[0058] Figure 3 It is a data interaction diagram provided according to an embodiment of the present disclosure.

[0059] In this embodiment, two users are taken as an example, respectively referred to as the first user and the second user. It is assumed that the first user needs to manage a first large data cluster (referred to as the first cluster) and the second user needs to manage a second large data cluster (referred to as the second cluster).

[0060] like Figure 3 As shown, the application layer 301 corresponds to two users. Taking the first user as an example, the first user sends a management request, which carries the identification information of the target big data cluster.

[0061] The platform 302 intercepts the management request sent by the user, and the platform can create a thread for each management request to process the corresponding management request. Assuming that the thread corresponding to the first user is called the first thread, the first thread can obtain identification information from the management request sent by the first user, and obtain corresponding metadata from the database based on the identification information, that is, obtain the metadata of the target big data cluster. The identification information of each candidate big data cluster and its corresponding metadata can be pre-recorded in the database.

[0062] After the first thread obtains the metadata of the target big data cluster, it can store it in thread local variables, which ensures that the metadata is accessible throughout the life cycle of the management request and is isolated from the management requests of other threads.

[0063] After the first thread obtains the metadata of the target big data cluster, it can also interact with the target big data cluster based on the metadata to manage the target big data cluster, such as the first thread forwarding the management request sent by the first user to the first cluster, receiving the return data from the first cluster, and returning it to the first user.

[0064] In combination with the above application scenarios, the present disclosure also provides the following embodiments:

[0065] Figure 4 is a schematic diagram according to a second embodiment of the present disclosure. This embodiment provides a management method for a big data cluster, which is applied to a unified platform. The unified platform is connected to multiple candidate big data clusters. The method includes:

[0066] 401. A user sends a management request to a unified platform, where the management request includes identification information of a target big data cluster to be managed.

[0067] 402. The unified platform intercepts the management request and obtains identification information of the target big data cluster from the management request.

[0068] 403. The unified platform obtains metadata of the target big data cluster from the database based on the identification information through the thread corresponding to the management request.

[0069] The unified platform may allocate a thread for each management request. For example, the platform may create multiple threads in advance, and allocate a created thread to each management request received. Alternatively, a thread may be created for the management request when a management request is received.

[0070] In this embodiment, the management request is processed by a thread corresponding to the management request, and different threads can be used to process different management requests, thereby achieving data isolation and improving processing security and efficiency.

[0071] Furthermore, the thread may obtain metadata from the database based on the identification information.

[0072] The identification information of each candidate big data cluster and its corresponding metadata are pre-recorded in the database, so that the metadata of the target big data cluster can be obtained based on the identification information of the target big data cluster.

[0073] In this embodiment, by obtaining metadata from a database based on identification information, metadata can be obtained simply and efficiently.

[0074] 404. The unified platform stores the metadata in a local variable of the thread.

[0075] After the thread obtains the metadata of the target big data cluster from the database, it can store it in the thread's local variables.

[0076] In this embodiment, by storing the metadata in thread local variables, it is convenient to read the metadata locally later, thereby improving data reading efficiency. In addition, metadata isolation can be achieved to improve data security.

[0077] 405. A unified platform manages the target big data cluster based on the metadata.

[0078] For example, if the target big data is a Hadoop cluster, the thread can manage the Hadoop cluster based on the metadata, send the management request sent by the user to the Hadoop cluster, and send the return data corresponding to the management request obtained by the Hadoop cluster to the user.

[0079] For example, if the management request is used to modify the Yarn queue configuration in the Hadoop cluster, after sending the management request to the Hadoop cluster, the Hadoop cluster completes the modification of the Yarn queue configuration based on the management request and returns the modified configuration result. Yarn (Yet Another Resource Negotiator) is a resource management and task scheduling framework in the Hadoop cluster. It effectively allocates resources in the cluster, such as the central processing unit (CPU), memory, etc., through a queue mechanism.

[0080] In addition, after completing the management of the target big data cluster, the thread local variables can be deleted, which can delete the corresponding metadata and effectively prevent memory leaks.

[0081] In some embodiments, the method may further include:

[0082] 406. The unified platform interacts with a lightweight management tool corresponding to the target big data cluster based on the metadata.

[0083] Among them, a lightweight management tool is, for example, Ambari, which is an open source cluster management tool.

[0084] Taking the target big data as a Hadoop cluster as an example, in some cases, Ambari corresponding to the Hadoop cluster can also be accessed based on the metadata of the Hadoop cluster.

[0085] Taking the Hadoop cluster as an example, managers can deploy, manage and monitor various components in the Hadoop cluster through the Web interface of Ambari corresponding to the Hadoop cluster. To this end, the Ambari corresponding to the Hadoop cluster can save relevant management information of the Hadoop cluster. In some cases, if these relevant management information are needed, they can be obtained from the Ambari corresponding to the Hadoop cluster, such as a thread sending a query request to Ambari based on metadata and receiving the query results sent by Ambari.

[0086] In this embodiment, the corresponding lightweight management tool is accessed based on the metadata of the target big data cluster, and relevant information can be obtained from the lightweight management tool. Cross-cluster management can be performed based on the relevant information in the lightweight management tools of different big data clusters, which facilitates cross-cluster resource scheduling and data sharing.

[0087] Figure 5This is a schematic diagram according to the third embodiment of the present disclosure. This embodiment provides a big data cluster management device, which is applied to a unified platform, wherein the unified platform is connected to multiple candidate big data clusters, and the device 500 includes: an interception module 501 , an acquisition module 502 and a management module 503 .

[0088] The interception module 501 is used to intercept management requests for a target big data cluster among the multiple candidate big data clusters; the management request contains identification information of the target big data cluster; the acquisition module 502 is used to acquire metadata of the target big data cluster based on the identification information; the management module 503 is used to manage the target big data cluster based on the metadata.

[0089] In this embodiment, multiple candidate big data clusters are managed based on a unified platform, which can achieve centralized, unified and efficient management, reduce management difficulty and cost, and thus improve management effect.

[0090] In some embodiments, the acquisition module 502 is further used to:

[0091] The metadata of the target big data cluster is acquired based on the identification information using a thread corresponding to the management request; wherein different management requests correspond to different threads.

[0092] In this embodiment, the management request is processed by a thread corresponding to the management request, and different threads can be used to process different management requests, thereby achieving data isolation and improving processing security and efficiency.

[0093] In some embodiments, the apparatus 500 further includes:

[0094] A storage module is used to store the metadata in a local variable of the thread.

[0095] In this embodiment, by storing the metadata in thread local variables, it is convenient to read the metadata locally later, thereby improving data reading efficiency. In addition, metadata isolation can be achieved to improve data security.

[0096] In some embodiments, the acquisition module 502 is further used to:

[0097] Acquire metadata of the target big data cluster from a database based on the identification information;

[0098] The database records identification information and metadata of each candidate big data cluster.

[0099] In this embodiment, by obtaining metadata from a database based on identification information, metadata can be obtained simply and efficiently.

[0100] In some embodiments, the apparatus 500 further includes:

[0101] The interaction module is used to interact with a lightweight management tool corresponding to the target big data cluster based on the metadata of the target big data cluster.

[0102] In this embodiment, the corresponding lightweight management tool is accessed based on the metadata of the target big data cluster, and relevant information can be obtained from the lightweight management tool. Cross-cluster management can be performed based on the relevant information in the lightweight management tools of different big data clusters, which facilitates cross-cluster resource scheduling and data sharing.

[0103] Figure 6 is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides a big data system, such as Figure 6 As shown, the system 600 includes: a unified platform 601 and a plurality of candidate big data clusters 602a-602n.

[0104] Each candidate big data cluster is connected to the unified platform, and the unified platform is as shown in any of the above embodiments.

[0105] In this embodiment, multiple candidate big data clusters are managed based on a unified platform, which can achieve centralized, unified and efficient management, reduce management difficulty and cost, and thus improve management effect.

[0106] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0107] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0108] It is understandable that, unless otherwise specified, the order of the steps in the process indicates that the timing relationship between these steps is not limited.

[0109] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0110] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0111] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0112] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0113] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a management method for a big data cluster. For example, in some embodiments, the management method for a big data cluster may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the management method for a big data cluster described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a management method for a big data cluster in any other appropriate manner (e.g., by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable information recommendation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0120] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A management method for big data clusters, applied to a unified platform, wherein the unified platform is connected to a plurality of candidate big data clusters, the method comprising: intercepting a management request for a target big data cluster among the multiple candidate big data clusters; The management request includes identification information of the target big data cluster; Based on the identification information, obtaining metadata of the target big data cluster; Based on the metadata, the target big data cluster is managed.

2. The method according to claim 1, wherein: The acquiring metadata of the target big data cluster based on the identification information includes: The metadata of the target big data cluster is acquired based on the identification information using a thread corresponding to the management request; wherein different management requests correspond to different threads.

3. The method according to claim 2, further comprising: The metadata is stored in local variables of the thread.

4. The method according to claim 1, wherein: The acquiring metadata of the target big data cluster based on the identification information includes: Acquire metadata of the target big data cluster from a database based on the identification information; The database records identification information and metadata of each candidate big data cluster.

5. The method according to claim 1, further comprising: Based on the metadata of the target big data cluster, interact with a lightweight management tool corresponding to the target big data cluster.

6. A big data cluster management device, applied to a unified platform, the unified platform is connected to multiple candidate big data clusters, the device comprises: An interception module, used to intercept a management request for a target big data cluster among the multiple candidate big data clusters; The management request includes identification information of the target big data cluster; An acquisition module, used to acquire metadata of the target big data cluster based on the identification information; A management module is used to manage the target big data cluster based on the metadata.

7. The device according to claim 6, wherein: The acquisition module is further used for: The metadata of the target big data cluster is acquired based on the identification information using a thread corresponding to the management request; wherein different management requests correspond to different threads.

8. The apparatus according to claim 7, further comprising: A storage module is used to store the metadata in a local variable of the thread.

9. The device according to claim 6, wherein: The acquisition module is further used for: Acquire metadata of the target big data cluster from a database based on the identification information; The database records identification information and metadata of each candidate big data cluster.

10. The apparatus according to claim 6, further comprising: The interaction module is used to interact with a lightweight management tool corresponding to the target big data cluster based on the metadata of the target big data cluster.

11. A big data system, comprising: Multiple candidate big data clusters; A unified platform connected to the plurality of candidate big data clusters; The unified platform is as described in any one of claims 6-10.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

14. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images