Automatic deployment method, system and equipment of big data platform and storage medium

Abstract

The invention provides an automatic deployment method, system and device of a big data platform and a storage medium, and relates to the technical field of computer application. The automatic deployment method comprises the following steps: reading an initial service configuration template of an initial service; generating a configuration file according to the initial service configuration template and initializing the cluster; obtaining a target service in the initial service and an initial service configuration template corresponding to the target service; modifying an initial service configuration template corresponding to the target service according to user-defined information of the user to obtain a second service configuration file; and generating a second format configuration file according to the second service configuration file, and deploying the target service according to the second format configuration file and the target node. By adopting the deployment method provided by the invention, the service configuration file corresponding to the target service can be directly obtained on the basis of the selected target service according to the initial service configuration template compatible with various components and the self-defined information, so that rapid deployment of the new service is realized.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, and more specifically, to an automated deployment method, system, device, and storage medium for a big data platform. Background Technology

[0002] In the fintech sector, financial institutions and enterprises have accumulated massive amounts of data resources as their businesses expand and deepen. This data includes customer information, transaction records, market data, risk assessments, and many other aspects, making it extremely valuable to financial institutions. However, traditional data processing and analysis methods are no longer sufficient to meet the demands of processing such massive amounts of data, necessitating the development of more advanced big data platforms to address this challenge.

[0003] In building big data platforms, developers face not only the challenge of adapting to various commercial hardware and Linux operating system versions, but also more complex distributed architectures and potential cross-dependencies between components. Against this backdrop, automated deployment solutions for big data platforms have become essential. Currently, relatively mature big data platforms include TDH, CDH, and Ambari. However, these platforms suffer from low customization levels and limited support for component versions. Furthermore, TDH and CDH platforms are heavily commercialized, requiring enterprises to bear licensing and procurement costs, as well as subsequent high technical maintenance expenses, resulting in significant overhead.

[0004] In the current big data field, platforms such as TDH, CDH, and Ambari have formed relatively mature application systems. However, these platforms still have several areas for optimization: First, their customization flexibility is insufficient, making it difficult to meet the deep adaptation needs of enterprises' differentiated business scenarios; second, their component version support system is relatively simple, resulting in limitations in technology iteration and compatibility; third, platforms such as TDH and CDH have prominent commercial attributes, requiring enterprises to bear not only licensing and procurement costs, but also subsequent technical maintenance costs, leading to a high usage expense and an overall imbalance in the return on investment. Summary of the Invention

[0005] To address the problems, or at least some of the problems, existing technologies described above, this invention provides an automated deployment method, system, storage medium, device, and computer program product for a big data platform. By abstracting an initial service configuration template compatible with various components, the system allows for the deployment of new services by directly obtaining the corresponding service configuration file based on the selected target service, according to the initial service configuration template and custom information. This helps developers quickly build a big data cloud-native platform.

[0006] According to a first aspect of the present invention, an embodiment of the present invention provides an automated deployment method for a big data platform. The method includes: acquiring node information in a cluster and deploying a metadata storage engine; reading an initial service configuration template of an initial service from an initial installation package; generating a first format configuration file that meets service requirements based on the initial service configuration template and storing the first format configuration file in the metadata storage engine; initializing the cluster based on the first format configuration file and node information; acquiring a target service in the initial service and an initial service configuration template corresponding to the target service; acquiring a role corresponding to the target service and a mapping relationship between the role and the target node; displaying the initial service configuration template corresponding to the target service on a display interface, modifying the initial service configuration template corresponding to the target service based on user-defined information to obtain a second service configuration file; distributing the installation package of the target service to the target node, generating a second format configuration file of the target service based on the second service configuration file, and deploying the target service based on the second format configuration file and the target node.

[0007] According to the above embodiments of the present invention, by abstracting an initial service configuration template that is compatible with various components, obtaining the initial service configuration templates of all initial services based on the initial installation package and performing cluster initialization, the deployment of new services can be achieved by directly obtaining the service configuration file corresponding to the target service based on the selected target service, according to the initial service configuration template and custom information. This helps developers quickly build a big data cloud-native platform, effectively reducing the daily operation and maintenance burden of developers and improving resource utilization.

[0008] In some embodiments of the present invention, the automated deployment method further includes: obtaining an extension service package and an extension service configuration file corresponding to the extension service; generating a third-format configuration file that meets the service requirements based on the extension service configuration file; obtaining the role corresponding to the extension service; and deploying the extension service based on the third-format configuration file and the role corresponding to the extension service.

[0009] In some embodiments of the present invention, the initial service configuration template includes: the name, version number, deployment package name and corresponding MD5 value of the initial service, the role name, role start command, role stop command, role running log path, minimal configuration information of the role, as well as the service dependencies and basic monitoring template information of the service.

[0010] In some embodiments of the present invention, the big data platform includes: a metadata management component, a data acquisition component, a data storage component, a resource scheduling component, a computing engine, a task scheduling component, a permission management component, and a service monitoring component.

[0011] According to the embodiments described above, the big data ecosystem possesses high compatibility, covering, but not limited to, data acquisition, data storage, resource scheduling, computing engines, task scheduling, access control, and service monitoring. It comprehensively covers the big data processing flow, enabling seamless collaboration among various components. Furthermore, this compatibility allows such a big data ecosystem to flexibly adapt to different hardware and software environments, meeting diverse enterprise needs.

[0012] According to a second aspect of the present invention, an automated deployment system for a big data platform is provided. The automated deployment system includes: a cluster initialization module, configured to perform the following steps: acquiring node information in the cluster and deploying a metadata storage engine; reading an initial service configuration template of an initial service from an initial installation package; generating a first format configuration file conforming to service requirements based on the initial service configuration template, and storing the first format configuration file in the metadata storage engine; initializing the cluster based on the first format configuration file and node information; a target service acquisition module, configured to acquire a target service in the initial service and an initial service configuration template corresponding to the target service, and acquire the role corresponding to the target service and the mapping relationship between the role and the target node; a configuration file acquisition module, configured to display the initial service configuration template corresponding to the target service on a display interface, and modify the initial service configuration template corresponding to the target service according to user-defined information to obtain a second service configuration file; and a target service deployment module, configured to distribute the installation package of the target service to the target node, generate a second format configuration file of the target service based on the second service configuration file, and deploy the target service based on the second format configuration file and the target node.

[0013] According to the above embodiments of the present invention, by abstracting an initial service configuration template that is compatible with various components, obtaining the initial service configuration templates of all initial services based on the initial installation package and performing cluster initialization, the deployment of new services can be achieved by directly obtaining the service configuration file corresponding to the target service based on the selected target service, according to the initial service configuration template and custom information. This helps developers quickly build a big data cloud-native platform, effectively reducing the daily operation and maintenance burden of developers and improving resource utilization.

[0014] In some embodiments of the present invention, the automated deployment system further includes a service extension module, configured to perform the following steps: obtaining an extension service package and an extension service configuration file corresponding to the extension service; generating a third-format configuration file that meets the service requirements based on the extension service configuration file; obtaining the role corresponding to the extension service; and deploying the extension service based on the third-format configuration file and the role corresponding to the extension service.

[0015] In some embodiments of the present invention, the initial service configuration template includes: the name, version number, deployment package name and corresponding MD5 value of the initial service, the role name, role start command, role stop command, role running log path, minimal configuration information of the role, as well as the service dependencies and basic monitoring template information of the service.

[0016] In some embodiments of the present invention, the big data platform includes: a metadata management component, a data acquisition component, a data storage component, a resource scheduling component, a computing engine, a task scheduling component, a permission management component, and a service monitoring component.

[0017] According to the embodiments described above, the big data ecosystem possesses high compatibility, covering, but not limited to, data acquisition, data storage, resource scheduling, computing engines, task scheduling, access control, and service monitoring. It comprehensively covers the big data processing flow, enabling seamless collaboration among various components. Furthermore, this compatibility allows such a big data ecosystem to flexibly adapt to different hardware and software environments, meeting diverse enterprise needs.

[0018] According to a third aspect of the present invention, an embodiment of the present invention provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, cause a computer to perform the following operations: the operations include the steps included in the automated deployment method of the big data platform as described in any of the above embodiments.

[0019] According to a fourth aspect of the present invention, an embodiment of the present invention provides a computer device including a memory and a processor, wherein the memory is used to store one or more computer-readable instructions, wherein the one or more computer-readable instructions, when executed by the processor, can realize the automated deployment method of the big data platform as described in any of the above embodiments.

[0020] According to a fifth aspect of the present invention, an embodiment of the present invention provides a computer program product including a computer program, which, when executed by a processor, implements the automated deployment method of a big data platform as described in any of the above embodiments.

[0021] As described above, the automated deployment method, system, storage medium, device, and computer program product of the big data platform provided by the embodiments of the present invention abstracts an initial service configuration template that is compatible with various components, obtains the initial service configuration templates of all initial services based on the initial installation package, and performs cluster initialization. Thus, subsequent deployment of new services can be achieved by directly obtaining the service configuration file corresponding to the selected target service based on the initial service configuration template and custom information. This helps developers quickly build a big data cloud-native platform, effectively reduces the daily operation and maintenance burden of developers, and improves resource utilization. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the automated deployment method of a big data platform according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a schematic diagram of a configuration interface according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the architecture of an automated deployment system for a big data platform according to Embodiment 3 of the present invention. Detailed Implementation

[0025] The various aspects of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Well-known modules, units, and their connections, links, communications, or operations are not shown or described in detail. Furthermore, the described features, architectures, or functions can be combined in any way in one or more embodiments. Those skilled in the art should understand that the various embodiments described below are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. It will also be readily understood that the modules, units, or processing methods in the embodiments described herein and shown in the accompanying drawings can be combined and designed in various different configurations.

[0026] The following is a brief explanation of the terminology used in this text.

[0027] MD5: Message-Digest Algorithm 5, a hash algorithm.

[0028] MySQL: A relational database management system.

[0029] JSON: JavaScript Object Notation, a lightweight data interchange format.

[0030] HBase: Hadoop Database, a distributed, columnar, NoSQL database built on the Hadoop Distributed File System (HDFS) for unstructured data.

[0031] Yum: Yellowdog Updater, Modified. An open-source package management tool based on RPM (Red Hat Package Manager), primarily used to automate the installation, updating, uninstallation, and dependency management of software packages in Linux systems.

[0032] HostName: The hostname is a string name that identifies a computer or node in a network and is used to distinguish different devices in a network environment.

[0033] NTP: Network Time Protocol, a standard protocol for synchronizing the time of devices in a network. It keeps the clocks of different devices (such as servers, routers, and clients) consistent by transmitting timestamp information across the network.

[0034] tar archive: an archive file format.

[0035] JAR file: An archive file format for the Java platform.

[0036] generateServiceCommand: A function or tool used to generate service startup commands.

[0037] CommandActor: A component responsible for executing commands in an automated deployment system or service orchestration framework.

[0038] Prometheus: An open-source system monitoring and alerting toolkit.

[0039] Grafana: An open-source graphical visualization tool.

[0040] API: Application Programming Interface.

[0041] [Example 1]

[0042] Figure 1 This is a flowchart illustrating the automated deployment method of a big data platform according to Embodiment 1 of the present invention.

[0043] The big data platform includes, but is not limited to, one or more of the following components: metadata management component, data acquisition component, data storage component, resource scheduling component, computing engine, task scheduling component, permission management component, and service monitoring component. For example, the storage and resource scheduling layer includes: HDFS distributed file system, YARN resource manager, and ZooKeeper distributed coordination service; the data processing and computing layer includes: Kafka streaming data platform, Hive data warehouse, Flink real-time computing engine, and Spark distributed computing framework; the storage and retrieval layer includes: HBase distributed database, ElasticSearch search engine, and Doris analytical database; and the service access layer includes: Kyubi multi-engine interaction gateway, etc. The built-in components of this invention have the latest recommended versions. At the same time, users are allowed to independently implement version selection and adaptation operations according to the specifications established by this invention, thereby ensuring the user's autonomy in version adaptation.

[0044] This big data platform incorporates a compatible and universal big data ecosystem, including but not limited to data acquisition, data storage, resource scheduling, computing engines, task scheduling, access control, and service monitoring. This comprehensive coverage of the big data processing workflow enables seamless collaboration among various components, forming a complete solution system. Furthermore, this compatibility allows this big data ecosystem to flexibly adapt to different hardware and software environments, meeting diverse enterprise needs.

[0045] In some implementations, users integrate or upgrade big data components through configuration to ensure the sustainability and scalability of the cluster.

[0046] like Figure 1 As shown, in Embodiment 1 of the present invention, the automated deployment method of the big data platform may include at least the following steps S11, S12, S13, S14, S15, S16, S17 and S18, which are described in detail below.

[0047] In step S11, node information in the cluster is obtained, and a metadata storage engine is deployed. In some implementations, the metadata storage engine may be, for example, MySQL.

[0048] In step S12, the initial service configuration template of the initial service in the initial installation package is read.

[0049] In some implementations, the initial service configuration template includes, but is not limited to: the name, version number, deployment package name and corresponding MD5 value of the initial service, the role name, role start command, role stop command, role running log path, the minimum configuration information of the role, as well as the dependencies between services and the basic monitoring template information of the service.

[0050] In some implementations, the initial service configuration template is the default service_ddl.json configuration file, and each component has its corresponding service_ddl.json file (i.e., a Service Definition Language JSON file or a Service DDL configuration file). Taking HBase as an example, its corresponding service_ddl.json file format is as follows:

[0051]

[0052]

[0053]

[0054] In step S13, a first format configuration file that meets the service requirements is generated according to the initial service configuration template, and the first format configuration file is stored in the metadata storage engine.

[0055] In some implementations, the metadata of each embedded service, `service_ddl.json`, is read, and an XML-formatted configuration file is generated according to the requirements of different services, serving as the primary configuration file. This allows for later support for, for example... Figure 2 The relevant configuration interface shown can be customized and saved, and then persisted to MySQL.

[0056] In this embodiment, the relationship between services and configurations is one-to-one. Each service generates a default configuration file based on the official minimal parameters. The configuration file model is displayed in key-value pair format. Here, the key mainly refers to the name of the parameter, and the value is the corresponding parameter value. Furthermore, parameter values ​​are categorized by the number of values ​​(single value and compound value, separated by commas); by whether the value is required (mandatory and optional); and by whether the value crosses services (inter-service and intra-service).

[0057] In one exemplary implementation, the values ​​of the parameters are displayed on the configuration interface in three forms, including but not limited to the following three:

[0058] 1) Edit box format: Allows users to flexibly customize and assemble configuration values.

[0059] 2) Switch type: Applicable to multiple combined parameters. Multiple parameters are configured by turning them on or off. These multiple parameters often need to be configured together and have an interlocking relationship, that is, the parameters exist simultaneously or are hidden. For example, when turned on, multiple interlocking parameters will be automatically displayed, and when turned off, the interlocking parameters will be automatically hidden.

[0060] 3) Sliding axis type: mainly for numerical data, which makes it easy to adjust the size of data values ​​by dragging.

[0061] In step S14, the cluster is initialized according to the first format configuration file and node information.

[0062] In step S15, the target service in the initial service and the initial service configuration template corresponding to the target service are obtained.

[0063] In step S16, the role corresponding to the target service and the mapping relationship between the role and the target node are obtained.

[0064] In step S17, the initial service configuration template corresponding to the target service is displayed on the display interface. The initial service configuration template corresponding to the target service is modified according to the user's custom information to obtain the second service configuration file.

[0065] In step S18, the installation package of the target service is distributed to the target node, and a second format configuration file of the target service is generated according to the second service configuration file. The target service is then deployed according to the second format configuration file and the target node.

[0066] The automated deployment method of the big data platform described in Embodiment 1 of this invention abstracts an initial service configuration template that is compatible with various components. Based on the initial installation package, the initial service configuration templates of all initial services are obtained and the cluster is initialized. Subsequently, based on the selected target service, the service configuration file corresponding to the target service can be obtained according to the initial service configuration template and custom information to realize the deployment of new services. This helps developers quickly build a big data cloud-native platform, effectively reduces the daily operation and maintenance burden of developers, and improves resource utilization.

[0067] In a further embodiment, the automated deployment method further includes: obtaining the extended service package and extended service configuration file corresponding to the extended service; generating a third-format configuration file that meets the service requirements based on the extended service configuration file; obtaining the role corresponding to the extended service; and deploying the extended service based on the third-format configuration file and the role corresponding to the extended service.

[0068] [Example 2]

[0069] Embodiment 2 of the present invention provides a method for the automated deployment and management of a big data platform. This method is implemented by a big data platform deployment system and includes at least the following steps:

[0070] Step 1: Cluster Initialization

[0071] When creating a cluster for the first time, first deploy the management node, Manager Service, and then perform the following steps:

[0072] S211. Determine the name of the cluster, which will serve as the unique identifier of the cluster.

[0073] S212. Configure the metadata storage engine, such as MySQL, for the Manage Service and configure the connection information. The metadata storage engine is used to store the cluster's metadata, such as the configuration and status information of each service.

[0074] S213. Verify the consistency of the embedded service's metadata, including but not limited to (component) version information and file MD5 values. Read the metadata of each embedded service, service_ddl.json, and generate an XML configuration file according to the requirements of different services, for example. service_ddl.json is the default configuration, which is an abstract configuration template compatible with various components. It was created during system installation and contains the necessary installation configurations for each component. Each component has its corresponding service_ddl.json file.

[0075] S214. After the initial installation, the service_ddl.json configuration file is written to the MySQL database so that custom configurations made on the page can be persisted to MySQL later.

[0076] S215. Execute the following cluster initialization process: (1) Specify the master node and disable the master node firewall; (2) Install the local offline Yum source on the master node; (3) Add basic dependency components; (4) Configure passwordless login; (5) Distribute the initialization resource package; (6) Create hadoop users and groups; (7) Disable the firewalls on all nodes; (8) Configure the Yum offline source on all nodes to point to the master node; (9) Configure the system environment on all nodes; (10) Configure the HostName on all nodes; (11) Configure the hosts mapping on all nodes; (12) Configure the NTP clock source on all nodes; (13) Install MySQL on the master node; (14) Initialize the database; (15) Disable the transparent big page on all nodes; (16) Configure the basic dependency library of the operating system.

[0077] Step 2: Add a new node

[0078] After the initial initialization, if you need to add more nodes, perform the following steps: (1) Configure passwordless login; (2) Distribute the initialization resource package; (3) Create hadoop users and groups; (4) Turn off the node firewall; (5) Configure the Yum source to point to the master node; (6) Configure the node system environment; (7) Configure the node HostName and distribute it to all nodes; (8) Configure the node hosts mapping and distribute it to all nodes; (9) Configure the node NTP clock source; (10) Turn off the node transparent big page; (11) Configure the operating system basic dependency library.

[0079] Step 3: Add a service

[0080] In this embodiment, the big data platform deployment system used to implement the automated deployment and management method of the big data platform serves as a deployment and management platform for big data. It supports a rich set of components, including but not limited to: HDFS, YARN, ZooKeeper, Kafka, Hive, Flink, Spark, HBase, ElasticSearch, Doris, and Kyubi. Furthermore, the system pre-integrates 32 service_ddl.json templates (i.e., the service_ddl.json of each service in step one, with default supported component versions embedded in the installation), covering the basic configuration parameters, dependencies, and deployment specifications of each component, providing a standardized input base for automated deployment. This template system adopts a key-value pair structured design, supports configurable parameter expansion, and can efficiently adapt to the cluster deployment needs of different business scenarios. Specific deployment steps include:

[0081] S231. Target service selection: Select the specified service / target service, such as Spark or Kafka.

[0082] S232. Assign the mapping relationship between all roles of the service (e.g., master service, meta metadata service) and cluster machines (physical nodes);

[0083] S233. Visual parameter configuration: The service configuration automatically loads the service_ddl.json template through the interface, and the user can customize the template configuration in this interface;

[0084] S234. Install and start the service, distribute the service installation package and instantiate the corresponding service configuration file according to the service_ddl.json template, and start the service.

[0085] In this embodiment, the DDL template is an abstract model of service information, encompassing the structured definition of the service version and its sub-services. It achieves semantic specification definitions for different services through standardized descriptions. The FTL template is the essential configuration information for service startup, containing the core parameter set required for service initialization and serving as a necessary configuration carrier to ensure normal service startup. Using FTL templates facilitates configuring various service parameters, while using DDL templates facilitates defining different services. Therefore, the system can automatically generate default service configurations (FTL templates) based on the FTL template engine. Optionally, it supports personalized parameter extensions / additions of personalized configurations and parameters within the DDL template. The FTL template has higher priority than the DDL template; that is, the template configuration reading order is FTL template first, then DDL template.

[0086] The pre-integrated templates include, but are not limited to, the service name, version number, deployment package name, and corresponding MD5 value; the role names, start and stop commands, log paths, minimal configuration information, service dependencies, and basic monitoring template information. By abstracting configuration templates compatible with various components and building a general, configurable, and scalable big data platform deployment system on top of these templates, service deployment, management, monitoring, and automated operation and maintenance can be integrated, significantly simplifying the operation and maintenance of big data components and helping developers quickly build stable, efficient, and elastically scalable big data cloud-native platforms.

[0087] Step 4: Add Extended Services

[0088] When the 32 pre-integrated component templates in the system cannot meet the requirements, developers can customize the necessary services based on the system's abstract template, that is, add custom extended services through a custom DDL+FTL template. The customization process includes: uploading the component package, configuring service parameters, and starting the service instance. Specific deployment steps include:

[0089] S241. Upload the service tar or jar package;

[0090] S242. Upload the service_ddl.json configuration file;

[0091] S243. Configuration file generation and verification;

[0092] S244. Assign service roles;

[0093] S245. Install and start the service.

[0094] Step 5: Component Management

[0095] When users issue various commands to the cluster through the front-end page, the Manager Service parses these front-end actions and generates executable commands based on the current component role. For example, when a user starts HBase and clicks the "Start" button, the backend calls the generateServiceCommand method; it queries the service role instance corresponding to the service, iterates through the role instance list, generates a local command, and finally notifies the commandActor to execute the command.

[0096] In this embodiment, the Worker is both the executor of the final command and the status manager of the partition it governs. The status information of the partitions governed by the Worker is stored in memory. The ManagerService sends heartbeat checks (pings) to each Worker at preset intervals; if a Worker does not respond within a specified time, the Master marks it as "failed". The Worker (worker node / process) monitors actions such as installation, service configuration, startup, and shutdown.

[0097] The above-described method for automated deployment and management of big data platforms according to Embodiment 2 of the present invention can achieve the following effects: (1) It simplifies the construction, deployment, management and monitoring process of big data clusters. (2) It provides a stable, efficient and scalable working environment, making data processing simple and fast. As a result, it can not only lower the entry threshold of big data clusters, but also significantly improve the stability and operating efficiency of the clusters. (3) By adopting an abstract modular design, the deployment of services is highly abstracted into: package distribution, service role allocation, role operation and maintenance including parameter configuration, role start and stop, service indicator monitoring including indicator collection, indicator storage, indicator display, indicator threshold configuration, establishment of dependencies between services and roles, etc., which not only facilitates code maintenance, but also allows users to combine different processing units according to their needs to achieve flexible functional expansion. (4) By integrating distributed computing and parallel processing technologies, it achieves high-efficiency computing performance and maximizes the utilization of hardware resources.

[0098] In one exemplary implementation, deploying a system based on the aforementioned big data platform can achieve intelligent monitoring and alerting services. Specifically, intelligent monitoring is implemented using Prometheus and Grafana components. Collection parameters are defined through a common JSON configuration template, and service metrics are periodically retrieved via the service's API. Intermittent time-series data collection achieves the purpose of service monitoring. Furthermore, an intuitive dashboard view allows users to clearly understand the health status of the cluster. Simultaneously, flexible alert settings are supported; alert rules can be customized as needed for CPU utilization, memory consumption, and network bandwidth, ensuring timely notification of anomalies and effectively guaranteeing stable system operation.

[0099] [Example 3]

[0100] Figure 3 This is a schematic diagram of the architecture of an automated deployment system for a big data platform according to Embodiment 3 of the present invention. The big data platform includes, but is not limited to: a metadata management component, a data acquisition component, a data storage component, a resource scheduling component, a computing engine, a task scheduling component, a permission management component, and a service monitoring component.

[0101] like Figure 3 As shown, the automated deployment system of the big data platform includes: a cluster initialization module 310, a target service acquisition module 320, a configuration file acquisition module 330, a target service deployment module 340, and a service expansion module 350.

[0102] The cluster initialization module 310 is used to perform the following steps: obtain node information in the cluster and deploy a metadata storage engine; read the initial service configuration template of the initial service in the initial installation package; generate a first format configuration file that meets the service requirements according to the initial service configuration template and store the first format configuration file in the metadata storage engine; initialize the cluster according to the first format configuration file and node information.

[0103] In some implementations, the initial service configuration template includes, but is not limited to, one or more of the following: the name, version number, deployment package name and corresponding MD5 value of the initial service, the role name, role start command, role stop command, role running log path, the minimum configuration information of the role, and the dependencies between services and the basic monitoring template information of the service.

[0104] The target service acquisition module 320 is used to acquire the target service in the initial service and the initial service configuration template corresponding to the target service, as well as the role corresponding to the target service and the mapping relationship between the role and the target node.

[0105] The configuration file acquisition module 330 is used to display the initial service configuration template corresponding to the target service on the display interface, and modify the initial service configuration template corresponding to the target service according to the user's custom information to obtain the second service configuration file.

[0106] The target service deployment module 340 is used to distribute the installation package of the target service to the target node, generate a second format configuration file of the target service according to the second service configuration file, and deploy the target service according to the second format configuration file and the target node.

[0107] The service extension module 350 is used to perform the following steps: obtain the extension service package and extension service configuration file corresponding to the extension service; generate a third-format configuration file that meets the service requirements based on the extension service configuration file; obtain the role corresponding to the extension service; and deploy the extension service based on the third-format configuration file and the role corresponding to the extension service.

[0108] The automated deployment system for the big data platform described in Embodiment 3 of this invention abstracts an initial service configuration template compatible with various components. Based on the initial installation package, it obtains the initial service configuration templates for all initial services and performs cluster initialization. Subsequently, it can directly deploy new services based on the selected target service, obtaining the corresponding service configuration file according to the initial service configuration template and custom information. This helps developers quickly build big data cloud-native platforms, effectively reducing their daily operational burden and improving resource utilization. Furthermore, functional expansion can be implemented as needed.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software combined with a hardware platform. Based on this understanding, all or part of the technical solution of the present invention that contributes to the background art can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0110] Correspondingly, embodiments of the present invention also provide a computer-readable storage medium storing computer-readable instructions or programs thereon. When executed by a processor, the computer-readable instructions or programs cause a computer to perform the following operations, which include the steps included in the automated deployment method of the big data platform described in any of the above embodiments, and will not be repeated here. The storage medium may include, for example, optical discs, hard disks, floppy disks, flash memory, magnetic tapes, etc.

[0111] Furthermore, embodiments of the present invention also provide a computer device including a memory and a processor. The memory is used to store one or more computer-readable instructions or programs, wherein when the one or more computer-readable instructions or programs are executed by the processor, they can realize the automated deployment method of the big data platform as described in any of the above embodiments. The computer device may be, for example, a server, a desktop computer, a laptop computer, a tablet computer, etc.

[0112] This invention also provides a computer program product including a computer program containing program code for executing the automated deployment method of the big data platform shown in the flowchart. When the computer program product runs on a computer system, the program code enables the computer system to implement the automated deployment method of the big data platform provided in this disclosure.

[0113] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Therefore, the scope of protection of the present invention should be determined by the claims.

Claims

1. An automated deployment method for a big data platform, characterized in that, The automated deployment method includes: Obtain node information from the cluster and deploy a metadata storage engine; Read the initial service configuration template of the initial service in the initial installation package; Generate a first format configuration file that meets the service requirements based on the initial service configuration template, and store the first format configuration file in the metadata storage engine; The cluster is initialized according to the first format configuration file and node information; Obtain the target service in the initial service and the initial service configuration template corresponding to the target service; Obtain the role corresponding to the target service and the mapping relationship between the role and the target node; The initial service configuration template corresponding to the target service is displayed on the display interface. The initial service configuration template corresponding to the target service is modified according to the user's custom information to obtain the second service configuration file. The installation package of the target service is distributed to the target node, and a second format configuration file of the target service is generated according to the second service configuration file. The target service is then deployed according to the second format configuration file and the target node.

2. The automated deployment method as described in claim 1, characterized in that, The automated deployment method also includes: Obtain the extension service package and extension service configuration file corresponding to the extension service; Generate a third-format configuration file that meets the service requirements based on the extended service configuration file; Obtain the role corresponding to the extended service; The extended service is deployed according to the third-format configuration file and the role corresponding to the extended service.

3. The automated deployment method as described in claim 1, characterized in that, The initial service configuration template includes: the name, version number, deployment package name and corresponding MD5 value of the initial service, the role name, role start command, role stop command, role running log path, the minimum configuration information of the role, as well as the service dependencies and basic monitoring template information of the service.

4. The automated deployment method as described in claim 1, characterized in that, The big data platform includes: metadata management component, data acquisition component, data storage component, resource scheduling component, computing engine, task scheduling component, permission management component, and service monitoring component.

5. An automated deployment system for a big data platform, characterized in that, The automated deployment system includes: The cluster initialization module is used to perform the following steps: Obtain node information from the cluster and deploy a metadata storage engine; Read the initial service configuration template of the initial service in the initial installation package; Generate a first format configuration file that meets the service requirements based on the initial service configuration template, and store the first format configuration file in the metadata storage engine; The cluster is initialized according to the first format configuration file and node information; The target service acquisition module is used to acquire the target service in the initial service and the initial service configuration template corresponding to the target service, as well as the role corresponding to the target service and the mapping relationship between the role and the target node; The configuration file acquisition module is used to display the initial service configuration template corresponding to the target service on the display interface, and modify the initial service configuration template corresponding to the target service according to the user's custom information to obtain the second service configuration file; The target service deployment module is used to distribute the installation package of the target service to the target node, generate a second format configuration file of the target service according to the second service configuration file, and deploy the target service according to the second format configuration file and the target node.

6. The automated deployment system as described in claim 5, characterized in that, The automated deployment system also includes a service extension module for performing the following steps: Obtain the extension service package and extension service configuration file corresponding to the extension service; Generate a third-format configuration file that meets the service requirements based on the extended service configuration file; Obtain the role corresponding to the extended service; The extended service is deployed according to the third-format configuration file and the role corresponding to the extended service.

7. The automated deployment system as described in claim 5, characterized in that, The initial service configuration template includes: the name, version number, deployment package name and corresponding MD5 value of the initial service, the role name, role start command, role stop command, role running log path, the minimum configuration information of the role, as well as the service dependencies and basic monitoring template information of the service.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that, The computer-readable instructions are executed by a processor to implement the automated deployment method as described in any one of claims 1-4.

9. A computer device comprising a memory and a processor, The memory stores computer-readable instructions, characterized in that, The processor executes the computer-readable instructions to implement the automated deployment method as described in any one of claims 1-4.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the automated deployment method as described in any one of claims 1-4.

Images