Big data componentization automatic deployment method and device, equipment and storage medium

Abstract

The invention provides an automatic deployment method and device for a big data component, equipment and a storage medium, and the method comprises the steps: obtaining deployment configuration information, determining a target configuration file template from a preset configuration file template library according to the big data component, carrying out the filling of the target configuration file template based on the deployment configuration information, and carrying out the automatic deployment of the big data component. Generating a target configuration file; creating a target cluster based on the cluster information, and deploying in the target cluster according to the target configuration file; in response to a configuration modification instruction, the configuration modification instruction comprises a target component and the modified configuration information, the configuration file template of the target component is filled according to the modified configuration information, and the target component is restarted. A user can select a required big data component on an interface to perform one-key automatic installation and deployment, and can dynamically adjust the cluster resource condition and add and delete the component at any time according to the service and the data volume of the user.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for the automated deployment of big data components. Background Technology

[0002] With the development and application of big data technology, enterprises are increasingly demanding more data storage, processing, and analysis capabilities. However, big data systems often encompass multiple components, involving various technologies and environmental configurations, making the deployment, management, and monitoring of the entire big data system quite complex. Traditional manual deployment and management methods are not only time-consuming and labor-intensive but also prone to errors, making it difficult to meet the needs of rapid iteration and expansion.

[0003] Existing automated deployment technologies primarily use configuration tools to automatically configure big data components, effectively simplifying the deployment process of big data systems and improving management efficiency. However, for different big data components, existing automated deployment technologies often require writing and maintaining different scripts, a cumbersome and complex process. Furthermore, if a component is added or removed, redeployment is necessary, resulting in a lack of flexibility. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for automated deployment of big data components, which solves the problem that existing automated deployment technologies require writing and maintaining different scripts for different big data components, resulting in a complex and inflexible process.

[0005] Firstly, this application provides a method for automated deployment of big data components, including:

[0006] Obtain deployment configuration information, which includes big data component and cluster information;

[0007] The big data component identifies the target configuration file template from the preset configuration file template library, fills the target configuration file template based on the deployment configuration information, and generates the target configuration file. The configuration file template library stores configuration file templates corresponding to the major big data components in advance.

[0008] A target cluster is created based on the cluster information, and the target cluster is deployed according to the target configuration file.

[0009] In response to a configuration modification command, which includes a target component and modified configuration information, the configuration file template of the target component is populated according to the modified configuration information, and the target component is restarted.

[0010] Optionally, in the method described above, the deployment configuration information further includes resource configuration and storage configuration. Based on the deployment configuration information, the target configuration file template is populated to generate the target configuration file, including:

[0011] Based on the correspondence between major data components and target configuration file templates, fill in the resource configuration and storage configuration corresponding to each major data component into the respective target configuration file templates;

[0012] The target configuration file templates are compiled and the target configuration file is generated.

[0013] Optionally, in the method described above, creating the target cluster based on the cluster information includes:

[0014] Based on the deployment configuration information, call the Kubernetes interface of the application software to create server cluster nodes;

[0015] The server cluster nodes are initialized.

[0016] Optionally, in the method described above, the deployment in the target cluster according to the target configuration file includes:

[0017] The configuration file reading script is invoked to read the target configuration file, and the configuration deployment is performed on each server cluster node based on the target configuration file;

[0018] Based on the network configuration of the target configuration file, the internal network configuration of the target cluster is performed. The internal network configuration is used for communication between the nodes of each server cluster.

[0019] Optionally, the method described above, before obtaining the deployment configuration information, further includes:

[0020] In response to the configuration parameters from the front end, the configuration parameters are parsed to obtain the data content;

[0021] The data content is verified, and big data components and cluster information are extracted from the verified data content.

[0022] Optionally, the method described above, after deploying in the target cluster according to the target configuration file, further includes:

[0023] Monitor each node and component of the target cluster according to preset indicators and obtain monitoring data;

[0024] Determine whether the monitored data exceeds the threshold of each preset indicator;

[0025] If so, push the alarm information to the management console.

[0026] Optionally, the method described above further includes:

[0027] The system collects the running status and installation information of the big data components of each node in the target cluster in real time, and then visualizes and displays the running status and installation information.

[0028] Secondly, this application provides an automated deployment device for big data components, comprising:

[0029] The configuration information acquisition module is used to acquire deployment configuration information, which includes big data component and cluster information;

[0030] The configuration file generation module is used to identify the target configuration file template from the preset configuration file template library according to the big data component, fill the target configuration file template based on the deployment configuration information, and generate the target configuration file. The configuration file template library stores configuration file templates corresponding to the big data components in advance.

[0031] The deployment module is used to create a target cluster based on the cluster information and deploy it in the target cluster according to the target configuration file.

[0032] The dynamic modification module is used to respond to configuration modification instructions, which include a target component and modified configuration information. The module populates the configuration file template of the target component according to the modified configuration information and restarts the target component.

[0033] Thirdly, this application provides an electronic device including a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-executable instructions to implement the big data component automated deployment method described in any one of the first aspects above.

[0034] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the big data component automated deployment method described in any one of the first aspects above.

[0035] Fifthly, this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the automated deployment method for big data components described in any of the first aspects above.

[0036] The automated deployment method, apparatus, device, and storage medium for big data components provided in this application acquire deployment configuration information, including big data component and cluster information; determine a target configuration file template from a preset configuration file template library based on the big data component; populate the target configuration file template based on the deployment configuration information to generate a target configuration file; the configuration file template library pre-stores configuration file templates corresponding to various big data components; create a target cluster based on the cluster information; deploy the system in the target cluster according to the target configuration file; respond to a configuration modification command, which includes the target component and modified configuration information, populate the configuration file template of the target component according to the modified configuration information, and restart the target component. Users can select the big data components they need for one-click automated installation and deployment on the interface, and can dynamically adjust cluster resources, as well as add and delete components, at any time according to their business and data volume. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 This is a schematic diagram illustrating an application scenario for the automated deployment method of big data components provided in the embodiments of this application;

[0039] Figure 2 A flowchart illustrating the automated deployment method for big data components provided in this application embodiment;

[0040] Figure 3 A schematic diagram of an automated deployment device for big data components provided in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for an automated deployment method of big data components, provided in an embodiment of this application.

[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] In related technologies, automatic configuration of big data components can be achieved through some configuration tools. However, for different big data components, existing automated deployment technologies often require writing and maintaining different scripts, which is tedious and complex. Once there is a need to add or delete components, redeployment is also required.

[0045] To address the aforementioned technical issues, this application aims to propose a method, apparatus, device, and storage medium for automated deployment of big data components. The core concept of this method is: by providing a simple and intuitive interface at the front end, users can select the deployment configuration information of the big data components they need on the interface, and then the server will automatically install and deploy them with one click based on the deployment configuration information. Furthermore, users can dynamically adjust the cluster resources and add or delete components at any time according to their business and data volume.

[0046] To better understand the solutions of the embodiments of this application, an application scenario involved in the embodiments of this application will be introduced below.

[0047] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario for the automated deployment method of big data components provided in the embodiments of this application, such as... Figure 1 As shown, the system includes a front-end 100, a server 200, and a Kubernetes (an open-source platform for managing containers) cluster 300. The front-end 100 primarily provides a selection interface for users, allowing them to confirm the big data components (e.g., Hadoop, Spark, Flink) and cluster information to be configured. Upon receiving the deployment configuration information, the server 200 determines the target configuration file template from a pre-defined template library based on the big data component, populates the target configuration file template with the deployment configuration information, generates the target configuration file, creates the target cluster in the Kubernetes cluster 300 based on the cluster information, and deploys it within the target cluster according to the target configuration file. Furthermore, the server 200 can also populate the target component's configuration file template with the modified configuration information according to the configuration modification command sent by the user from the front-end 100, and then restart the target component to update its deployment.

[0048] It should be noted that server 200 can also monitor preset metrics for each node and component of the target cluster in Kubernetes cluster 300, acquire monitoring data, and determine whether to issue alarms based on the monitoring data. Server 200 can also collect the running status and installation information of big data components on each node of the target cluster in Kubernetes cluster 300 in real time, and push the running status and installation information to front-end 100 for visualization.

[0049] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0050] Figure 2 A flowchart illustrating the automated deployment method for big data components provided in this application embodiment. Figure 2 As shown, the method in this embodiment includes:

[0051] S201: Obtain deployment configuration information, which includes big data component and cluster information.

[0052] The execution entity of this application embodiment can be a server or an automated deployment system for big data components within the server. The automated deployment system for big data components can be implemented through software.

[0053] Understandably, obtaining deployment configuration information is the starting point of the entire automated deployment process. Suppose a user wants to deploy a Hadoop cluster. The user can select Hadoop as the big data component through the front-end interface and input cluster information, such as the required number of nodes (e.g., 5 nodes), node specifications (e.g., 4 cores and 8GB of memory per node), network configuration, etc. This information constitutes the deployment configuration information.

[0054] It is understandable that, before obtaining deployment configuration information, the process may also include: responding to configuration parameters from the front end, parsing the configuration parameters to obtain data content; verifying the data content, and extracting big data components and cluster information from the verified data content.

[0055] In other words, before obtaining deployment configuration information, the system needs to respond to the configuration parameters passed from the front end and parse these parameters. The parsing process converts the configuration parameters entered or selected by the user on the front end into data content that the system can understand and process. After parsing the configuration parameters and obtaining the data content, the system needs to verify this data content. Unverified data may lead to configuration errors, wasted resources, or even system crashes. The verification process ensures that the configuration parameters provided by the user are valid, reasonable, and meet the system's requirements. Only verified data content can be used for subsequent configuration and deployment. Furthermore, verification allows for the early detection and correction of potential problems, ensuring a smooth deployment process.

[0056] S202: Based on the big data component, the target configuration file template is confirmed from the preset configuration file template library, and the target configuration file template is filled in based on the deployment configuration information to generate the target configuration file. The configuration file template library pre-stores configuration file templates corresponding to the big data components.

[0057] In this embodiment, using a preset configuration file template can standardize the configuration process, reduce human error, and improve configuration efficiency and accuracy. The big data component automated deployment system can find the corresponding configuration file template from the preset configuration file template library according to the big data component selected by the user, and then populate the template according to the deployment configuration information provided by the user to generate the final target configuration file.

[0058] For example, if a user selects the Hadoop component, the big data component automated deployment system will find the Hadoop configuration file template from the configuration file template library. The template may contain some placeholders, such as the number of nodes and memory size. The system will fill the template with the specific information provided by the user (5 nodes, 4 cores and 8GB of memory per node) to generate a specific Hadoop configuration file.

[0059] S203: Create a target cluster based on the cluster information, and deploy it in the target cluster according to the target configuration file.

[0060] In this step, a new target cluster can be created within a Kubernetes cluster based on the cluster information provided by the user. Then, the system uses the generated target configuration file to deploy big data components within this target cluster. For example, a new target cluster can be created within a Kubernetes cluster based on the cluster information provided by the user. Then, the system uses the generated target configuration file to deploy big data components within this target cluster.

[0061] S204: In response to a configuration modification instruction, the configuration modification instruction including a target component and modified configuration information, the configuration file template of the target component is filled according to the modified configuration information, and the target component is restarted.

[0062] In this embodiment, the automated deployment system for big data components can respond to configuration modification commands sent by users from the front end. Suppose a user wants to increase the number of nodes in the Hadoop cluster from 5 to 7. The user sends the configuration modification command through the front end. Upon receiving the command, the system updates the node count information in the Hadoop configuration file template and generates a new configuration file. Then, the system restarts the Hadoop components to run under the new configuration. Thus, the automated deployment system for big data components can flexibly adjust system configurations to meet dynamic user needs, ensuring system flexibility and scalability.

[0063] The automated deployment method for big data components provided in this embodiment obtains deployment configuration information, including big data component and cluster information; identifies a target configuration file template from a preset configuration file template library based on the big data component; populates the target configuration file template based on the deployment configuration information to generate a target configuration file; the configuration file template library pre-stores configuration file templates corresponding to various big data components; creates a target cluster based on the cluster information; and deploys the system in the target cluster according to the target configuration file. In response to a configuration modification command, which includes the target component and modified configuration information, the configuration file template of the target component is populated according to the modified configuration information, and the target component is restarted. This ensures that the big data components can be deployed and run efficiently and accurately according to user needs. Through standardized configuration file templates and a flexible configuration modification mechanism, deployment efficiency is improved, and the system's flexibility and maintainability are enhanced.

[0064] The technical solution for the above-mentioned automated deployment method of big data components will be described in detail below.

[0065] In one possible implementation, the automated deployment method for big data components provided in this embodiment includes resource configuration and storage configuration in the deployment configuration information. The target configuration file template is populated based on the deployment configuration information to generate the target configuration file, including: filling the resource configuration and storage configuration corresponding to each big data component into each target configuration file template according to the correspondence between the big data components and the target configuration file template; and summarizing the target configuration file templates to generate the target configuration file.

[0066] Understandably, resource and storage configurations are fundamental to the normal operation of big data components, ensuring that the system can allocate computing and storage resources reasonably according to user needs. Deployment configuration information includes not only big data component and cluster information, but also resource and storage configurations. Resource configuration typically involves computing resources (such as CPU, memory, etc.), while storage configuration involves data storage paths, storage types (such as local storage, distributed storage, etc.).

[0067] For example, when a user wants to deploy a Hadoop cluster and specifies that each node has 4 CPU cores and 8GB of memory, and that HDFS (Hadoop Distributed File System) is used for storage with the data storage path specified as " / data / hadoop", they can find Hadoop configuration file templates in the configuration file template library and fill in the user-provided resource configuration (4 CPU cores, 8GB memory) and storage configuration (HDFS, data storage path " / data / hadoop") into the template. These templates are then aggregated to generate a complete Hadoop configuration file. This configuration file will contain all the necessary information, such as the number of nodes, resource configuration, and storage configuration.

[0068] In this embodiment, the configuration file template is populated through resource configuration and storage configuration to ensure that all configuration information is correctly integrated, providing a complete configuration basis for subsequent deployment.

[0069] In one possible implementation, the method for automated deployment of big data components provided in this embodiment, which creates a target cluster based on the cluster information, includes: creating server cluster nodes by calling the interface of the application software Kubernetes according to the deployment configuration information; and initializing the server cluster nodes.

[0070] Understandably, the automated deployment system for big data components creates server cluster nodes by calling Kubernetes APIs based on the deployment configuration information provided by the user. These nodes may be virtual machines or physical machines, depending on the underlying infrastructure. These nodes will form a new Kubernetes cluster for deploying big data components.

[0071] Node initialization is a critical step in ensuring that server cluster nodes can function correctly and join the Kubernetes cluster. Uninitialized nodes may fail to run big data components properly and could even cause cluster instability. Therefore, the initialization process is essential for ensuring system stability and reliability. The initialization process typically includes installing necessary software packages, configuring the network, and setting security policies to ensure that nodes can function correctly and join the Kubernetes cluster.

[0072] For example, when a user wants to deploy a Hadoop cluster and specifies that 5 nodes are needed, the big data component automated deployment system will call the Kubernetes API to create 5 server cluster nodes. After creating the 5 Hadoop nodes, the system needs to initialize each node. This includes installing Docker as a container runtime environment, configuring the network to ensure communication between nodes, and setting firewall rules for security. Once initialization is complete, these nodes will be added to the Kubernetes cluster and become part of the cluster.

[0073] In this embodiment, creating nodes automatically can greatly reduce the complexity and error rate of manual operations, improve deployment efficiency and consistency, and ensure the stability and reliability of newly created nodes by performing necessary initialization operations, thus providing a guarantee for the smooth deployment and operation of big data components.

[0074] Furthermore, deployment in the target cluster based on the target configuration file may include: calling a configuration file reading script to read the target configuration file, and performing configuration deployment on each server cluster node based on the target configuration file; and performing internal network configuration of the target cluster based on the network configuration of the target configuration file, wherein the internal network configuration is used for communication between each server cluster node.

[0075] Understandably, the system invokes a configuration file reading script to read the generated target configuration file. Based on the read configuration content, it executes the corresponding configuration deployment operations on each server cluster node. This involves internal network configuration of the target cluster. Big data components typically require significant data transfer and task coordination across multiple nodes; good network configuration can significantly improve cluster performance and stability. Internal network configuration of the target cluster can include setting communication rules between server nodes, network topology, IP address allocation, etc., to ensure smooth communication between nodes within the cluster.

[0076] For example, when a big data component automated deployment system has generated a target configuration file containing Hadoop configurations, specifying the resource configuration, storage path, network settings, etc., for each node, this configuration file can be read by calling a configuration file reading script. Corresponding configuration operations can then be performed on each server node, such as installing Hadoop packages, setting HDFS storage paths, and configuring YARN resource management. Furthermore, the target configuration file may specify the IP address and network topology for each node. Based on this information, the system configures the network within the cluster, enabling the Hadoop nodes to communicate with each other via the specified IP addresses, ensuring smooth data transmission and task scheduling.

[0077] In this embodiment, by automatically reading the target configuration file and performing corresponding configuration operations on each node, human error is reduced, configuration consistency and deployment efficiency are improved, and the system is ensured to run as expected.

[0078] In one possible implementation, after the big data component automated deployment method provided in this embodiment is deployed in the target cluster according to the target configuration file, it further includes: monitoring each node and each component of the target cluster according to preset indicators and obtaining monitoring data; determining whether the monitoring data exceeds the threshold of each preset indicator; and if so, pushing alarm information to the management terminal.

[0079] Understandably, after the target cluster is successfully deployed, the system needs to continuously monitor each node and component within the cluster. Monitoring metrics can include CPU utilization, memory utilization, disk I / O, and network traffic. By monitoring these preset metrics, the system can understand the cluster's operational status in real time. For example, a node might have 70% CPU utilization, 60% memory utilization, and 80% HDFS storage utilization.

[0080] Furthermore, the acquired monitoring data can be analyzed to determine whether it exceeds preset thresholds. These preset thresholds are set based on system performance requirements and resource limitations; exceeding them may indicate system anomalies or the need for adjustments. For example, suppose the system sets CPU utilization thresholds at 80%, memory utilization at 75%, and HDFS storage utilization at 85%. The system compares real-time monitoring data to these thresholds. If a node's CPU utilization exceeds 80%, or its HDFS storage utilization exceeds 85%, the system considers these metrics to have exceeded the preset thresholds.

[0081] In this embodiment, alarm information typically includes the specific value of the abnormal indicator, the degree to which it exceeds the threshold, and possible causes, so that administrators can take timely measures to handle the situation. For example, suppose the CPU utilization of a node reaches 90%, exceeding the preset 80% threshold. The system will generate an alarm message, which may include: "Node X's CPU utilization is 90%, exceeding the preset threshold of 80%. Please check for abnormal task execution or unreasonable resource allocation." This alarm message will be pushed to the management terminal to remind operations and maintenance personnel to check and handle the issue.

[0082] In this embodiment, by setting reasonable thresholds, the system can promptly identify indicators that require attention, thereby avoiding resource overload and performance degradation.

[0083] In one possible implementation, the automated deployment method for big data components provided in this embodiment further includes: real-time collection of the running status and installation information of the big data components of each node in the target cluster, and visualization of the running status and installation information.

[0084] In this embodiment, after the target cluster is successfully deployed and starts running, the system needs to continuously collect the running status and installation information of the big data components on each node. Running status includes real-time performance metrics such as CPU utilization, memory utilization, disk I / O, and network traffic, while installation information includes software version, configuration file path, and dependent libraries. After collecting the running status and installation information of the big data components on each node, the system needs to visualize this information. Visualization typically uses charts, dashboards, and other methods to present complex data intuitively to users, facilitating understanding and analysis. For example, a line chart can be used to show the CPU utilization trend of each node, a pie chart can be used to show HDFS storage usage, and a table can be used to show the Hadoop version number and configuration file path of each node. Users can intuitively understand the cluster's running status and configuration through these visual charts.

[0085] In this embodiment, by collecting and displaying the operating status and installation information in real time, the operating status and installation information of the big data component can be collected and displayed efficiently and accurately, providing users with comprehensive monitoring and management tools to ensure the stable operation and high performance of the big data component.

[0086] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0087] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0088] Figure 3 This is a schematic diagram of an automated deployment device for big data components provided in an embodiment of this application. Figure 3 As shown, the automated deployment device for big data components includes:

[0089] Configuration information acquisition module 31 is used to acquire deployment configuration information, which includes big data component and cluster information;

[0090] The configuration file generation module 32 is used to identify the target configuration file template from the preset configuration file template library according to the big data component, fill the target configuration file template based on the deployment configuration information, and generate the target configuration file. The configuration file template library stores configuration file templates corresponding to the big data components in advance.

[0091] Deployment module 33 is used to create a target cluster based on the cluster information and deploy it in the target cluster according to the target configuration file;

[0092] The dynamic modification module 34 is used to respond to a configuration modification command, which includes a target component and modified configuration information, to fill the configuration file template of the target component according to the modified configuration information, and to restart the target component.

[0093] In one possible design, the deployment configuration information also includes resource configuration and storage configuration, and the configuration file generation module 32 is specifically used for:

[0094] Based on the correspondence between major data components and target configuration file templates, fill in the resource configuration and storage configuration corresponding to each major data component into the respective target configuration file templates;

[0095] The target configuration file templates are compiled and the target configuration file is generated.

[0096] In one possible design, deployment module 33 is specifically used for:

[0097] Based on the deployment configuration information, call the Kubernetes interface of the application software to create server cluster nodes;

[0098] The server cluster nodes are initialized.

[0099] In one possible design, deployment module 33 is specifically used for:

[0100] The configuration file reading script is invoked to read the target configuration file, and the configuration deployment is performed on each server cluster node based on the target configuration file;

[0101] Based on the network configuration of the target configuration file, the internal network configuration of the target cluster is performed. The internal network configuration is used for communication between the nodes of each server cluster.

[0102] In one possible design, the configuration information acquisition module 31 is also specifically used for:

[0103] In response to the configuration parameters from the front end, the configuration parameters are parsed to obtain the data content;

[0104] The data content is verified, and big data components and cluster information are extracted from the verified data content.

[0105] In one possible design, the automated deployment device for big data components also includes a monitoring module 35, which is specifically used for:

[0106] Monitor each node and component of the target cluster according to preset indicators and obtain monitoring data;

[0107] Determine whether the monitored data exceeds the threshold of each preset indicator;

[0108] If so, push the alarm information to the management console.

[0109] In one possible design, the automated deployment device for big data components also includes a data acquisition module 36, which is specifically used for:

[0110] The system collects the running status and installation information of the big data components of each node in the target cluster in real time, and then visualizes and displays the running status and installation information.

[0111] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0112] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0113] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for an automated deployment method of big data components, provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device of this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown in the diagram) a processor, a memory 41, and a computer program stored in the memory 41 and executable on at least one processor 40, which, when executing the computer program, implements the steps in any of the above method embodiments.

[0114] The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0115] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0116] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0117] In some embodiments, memory 41 may be an internal storage unit of an electronic device, such as the memory of the electronic device. In other embodiments, memory 41 may be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 41 may include both internal and external storage units of the electronic device. Memory 41 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 41 can also be used to temporarily store data that has been output or will be output.

[0118] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0119] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0120] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the aforementioned electronic device.

[0121] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0122] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0123] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0124] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for automated deployment of big data components, characterized in that, The method comprises the following steps: obtaining deployment configuration information, wherein the deployment configuration information comprises big data component and cluster information; confirming a target configuration file template from a preset configuration file template library according to the big data component, filling the target configuration file template based on the deployment configuration information, and generating a target configuration file, wherein the configuration file template library pre-stores configuration file templates corresponding to each big data component; creating a target cluster based on the cluster information, and deploying in the target cluster according to the target configuration file; in response to a configuration modification instruction, wherein the configuration modification instruction comprises a target component and modified configuration information, filling the configuration file template of the target component according to the modified configuration information, and restarting the target component.

2. The method of claim 1, wherein, The deployment configuration information further comprises resource configuration and storage configuration, and filling the target configuration file template based on the deployment configuration information to generate a target configuration file comprises the following steps: filling the resource configuration and the storage configuration corresponding to each big data component into each target configuration file template according to the correspondence between each big data component and the target configuration file template; summarizing each target configuration file template to generate a target configuration file.

3. The method of claim 1, wherein, The method of creating a target cluster based on the cluster information comprises the following steps: calling an interface of application software Kubernetes to create a server cluster node according to the deployment configuration information; initializing the server cluster node.

4. The method of claim 3, wherein, The method of deploying in the target cluster according to the target configuration file comprises the following steps: calling a configuration file reading script to read the target configuration file, and performing configuration deployment on each server cluster node based on the target configuration file; performing internal network configuration of the target cluster based on the network configuration of the target configuration file, wherein the internal network configuration is used for communication between each server cluster node.

5. The method of claim 1, wherein, Before the step of obtaining the deployment configuration information, the method further comprises the following steps: in response to a front-end configuration parameter, analyzing the configuration parameter to obtain data content; verifying the data content, and extracting big data components and cluster information from the data content that passes the verification.

6. The method according to any one of claims 1 to 5, characterized in that, After the step of deploying in the target cluster according to the target configuration file, the method further comprises the following steps: monitoring each node and each component of the target cluster according to a preset index to obtain monitoring data; judging whether the monitoring data exceeds a threshold of each preset index; if yes, pushing alarm information to a management end.

7. The method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps: collecting running states and installation information of big data components of each node in the target cluster in real time, and visually displaying the running states and the installation information.

8. A big data component automated deployment apparatus, characterized by, The method comprises the following steps: a configuration information obtaining module is configured to obtain deployment configuration information, wherein the deployment configuration information comprises big data component and cluster information; a configuration file generating module is configured to confirm a target configuration file template from a preset configuration file template library according to the big data component, fill the target configuration file template based on the deployment configuration information, and generate a target configuration file, wherein the configuration file template library pre-stores configuration file templates corresponding to each big data component; a deployment module, configured to create a target cluster based on the cluster information, and to deploy in the target cluster according to the target configuration file; a dynamic modification module, configured to, in response to a configuration modification instruction including a target component and modified configuration information, fill a configuration file template of the target component according to the modified configuration information, and restart the target component.

9. An electronic device, comprising: comprising: a processor, and a memory connected with the processor in communication; the memory stores computer-executed instructions; the processor executes the computer-executed instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores computer-executed instructions, and the computer-executed instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

Images