Data cluster deployment method and device, electronic equipment and storage medium

Abstract

The invention discloses a data cluster deployment method and device, electronic equipment and a storage medium, and relates to the technical field of computers.The data cluster deployment method comprises the steps that hardware configuration parameters of a virtual machine are obtained; based on the hardware configuration parameters, determining hardware key features of the virtual machine; wherein the hardware key features at least comprise processor features; based on the hardware key features, configuring component parameters of components of the to-be-deployed data cluster to obtain a configuration file; and finally, sending the configuration file to the virtual machine, so that the virtual machine deploys the to-be-deployed data cluster based on the configuration file. Namely, the configuration of the configuration file of the data cluster can be expanded according to the processor characteristics by determining the processor characteristics of the virtual machine, and the configuration of the configuration file considers the processor characteristics, so that the deployment of the data cluster is expanded based on the configuration file configured by the processor characteristics, and the performance of the data cluster can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for deploying a data cluster. Background Technology

[0002] With the rapid development of cloud platforms, deploying data clusters using cloud platform computing resources can meet the needs of data processing and analysis.

[0003] However, when deploying data clusters on cloud platforms, traditional big data cluster deployment tools often assume that virtual machines use default hardware configurations (such as defaulting to CPU (Central Processing Unit) based virtual machines). This assumption fails to determine the actual hardware configuration of the virtual machines, which may result in the inability to fully utilize the heterogeneous computing resources of the virtual machines, thus limiting the computing efficiency and task execution speed of the entire data cluster and failing to maximize the overall performance of the data cluster.

[0004] In summary, conventional technologies face the challenge of efficiently utilizing heterogeneous computing resources when deploying data clusters on cloud platforms, thus limiting the performance of the data clusters. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for deploying data clusters, in order to at least solve the problem in related technologies where it is difficult to utilize heterogeneous computing resources, which limits the performance of data clusters.

[0006] This application provides a method for deploying a data cluster, comprising: obtaining first configuration parameters of a virtual machine; wherein the first configuration parameters are set hardware configuration parameters of the virtual machine; determining key hardware features of the virtual machine based on the first configuration parameters; wherein the key hardware features include at least: processor features; configuring component parameters of components of the data cluster to be deployed based on the key hardware features to obtain a configuration file of the data cluster to be deployed; and sending the configuration file to the virtual machine so that the virtual machine deploys the data cluster to be deployed based on the configuration file.

[0007] This application also provides a data cluster deployment apparatus, comprising: an acquisition module for acquiring first configuration parameters of a virtual machine; wherein the first configuration parameters are set hardware configuration parameters of the virtual machine; a determination module for determining key hardware features of the virtual machine based on the first configuration parameters; wherein the key hardware features include at least: processor features; a configuration module for configuring component parameters of components of the data cluster to be deployed based on the key hardware features, thereby obtaining a configuration file of the data cluster to be deployed; and a deployment module for sending the configuration file to the virtual machine, so that the virtual machine deploys the data cluster to be deployed based on the configuration file.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described data cluster deployment methods.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data cluster deployment methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data cluster deployment methods.

[0011] This application obtains the hardware configuration parameters of a virtual machine; based on the hardware configuration parameters, it determines the key hardware characteristics of the virtual machine; wherein the key hardware characteristics include at least: processor characteristics; further, based on the key hardware characteristics, it configures the component parameters of the data cluster to be deployed, obtaining the configuration file of the data cluster to be deployed; finally, it sends the configuration file to the virtual machine, so that the virtual machine deploys the data cluster to be deployed based on the configuration file. In other words, by obtaining the hardware configuration parameters of the virtual machine and determining its processor characteristics, the configuration file of the data cluster can be configured according to the processor characteristics. Since the configuration file takes the processor characteristics into account, the deployment of the data cluster based on the configuration file configured with the processor characteristics can improve the performance of the data cluster. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A structural block diagram of a server device for a data cluster deployment method provided in an embodiment of this application;

[0014] Figure 2 A flowchart illustrating a data cluster deployment method provided in an embodiment of this application;

[0015] Figure 3 A flowchart illustrating a data cluster deployment method provided in another embodiment of this application;

[0016] Figure 4 This is a structural block diagram of a data cluster deployment device provided in an embodiment of this application. Detailed Implementation

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The data cluster deployment method embodiments provided in this application can be executed on server devices or similar computing devices. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a data cluster deployment method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0021] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data cluster deployment method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0023] This application provides an embodiment of a data cluster deployment method applied to the aforementioned server equipment. The method is described in detail below, following the execution flow of the data cluster deployment method. Figure 2 As shown, the method includes the following steps S202-S208:

[0024] S202, Obtain the first configuration parameters of the virtual machine; where the first configuration parameters are the hardware configuration parameters of the virtual machine that are set.

[0025] The first configuration parameter refers to the theoretical configuration information of the virtual machine. The first configuration parameter may include, but is not limited to, processor type (such as CPU, GPU (Graphics Processing Unit), number of cores, memory size, storage type (such as HDD (Hard Disk Drive) or SSD (Solid State Drive)).

[0026] It should be noted that the virtual machine's primary configuration parameters can be queried through the cloud platform's metadata service or API (Application Programming Interface). For example, a script can be used to send a request to a specified metadata service or API, which can then return the virtual machine's theoretical configuration information (i.e., the primary configuration parameters). The virtual machine's theoretical configuration information may include: the virtual machine's processor model, number of cores, frequency, storage type, network bandwidth limits, etc.

[0027] In some embodiments, when retrieving the first configuration parameter using an API, the API call may encounter various abnormal situations, such as network latency, server errors, and API rate limiting. Therefore, stability and reliability can be improved based on an exception handling mechanism. For example, if an API call returns an error code 4XX, exponential backoff retries can be automatically enabled (the initial interval can be 2 seconds, and the maximum number of retries can be 5). If the API still does not return data normally after reaching the maximum number of retries, an error will be reported.

[0028] S204, Based on the first configuration parameters, determine the key hardware characteristics of the virtual machine; wherein, the key hardware characteristics include at least: processor characteristics;

[0029] Specifically, the processor's specific model, number of cores and threads, and cache level and size can be extracted from the first configuration parameters to determine the processor's characteristics.

[0030] In an exemplary embodiment, the method for determining the processor characteristics includes: obtaining a second configuration parameter of the virtual machine; wherein the second configuration parameter is the actual hardware configuration parameter of the virtual machine detected by the virtual machine's hardware detection script; comparing the first configuration parameter with the second configuration parameter to determine the processor type of the virtual machine's processor; and determining the processor characteristics based on the processor type.

[0031] The hardware detection script is a program or set of instructions used to detect the hardware configuration parameters of a virtual machine. The second configuration parameter refers to the actual configuration information of the virtual machine obtained based on the hardware detection script inside the virtual machine. Comparing the first and second configuration parameters aims to verify whether the actual configuration information of the virtual machine matches the theoretical configuration information provided by the cloud platform.

[0032] In some embodiments, when comparing the first configuration parameter and the second configuration parameter, the comparison can be carried out separately according to the parameter type, such as comparing processor type, number of cores, memory size, storage type, etc., to obtain key hardware characteristics. By comparing hardware configuration parameters obtained through different methods, the accuracy of key hardware characteristics can be effectively improved.

[0033] In some embodiments, the processor type in the first configuration parameter can be compared with the processor type in the second configuration parameter. If the processor types are inconsistent, it can be determined whether the driver is not installed. Further, the virtual machine operating system version is obtained, and the corresponding driver is retrieved and installed. After successful installation, the hardware detection script is used again to obtain the new second configuration parameter. If the driver installation fails, or the processor type is still inconsistent after installation, the virtual machine's processor type is automatically identified as CPU.

[0034] In some embodiments, after determining the processor type, the processor type can be directly used as a processor feature. Of course, other processor features can also be further analyzed, such as the number of processor cores and threads, clock frequency, supported memory types and frequencies, etc., and these other processor features, along with the processor type, can be used together as processor features.

[0035] In the above embodiments, determining processor characteristics based on processor type is a crucial prerequisite for achieving efficient deployment and operation of the data cluster. By leveraging processor characteristics, component parameters can be configured specifically, fully utilizing hardware resources and effectively improving the computing efficiency and application performance of virtual machines.

[0036] In an exemplary embodiment, the first configuration parameter includes: a first attribute parameter of the processor; the second configuration parameter includes: a second attribute parameter of the processor; comparing the first configuration parameter with the second configuration parameter to determine the processor type of the virtual machine's processor includes: comparing the first attribute parameter and the second attribute parameter, and if both the first attribute parameter and the second attribute parameter indicate that the processor is an accelerated processor, then the processor type is determined to be an accelerated processor; otherwise, the processor type is determined to be a general-purpose processor.

[0037] The first attribute parameter refers to processor information obtained directly from the cloud platform's API interface or hardware description file, which may include the processor model, the number of physical cores, etc. The second attribute parameter refers to processor information obtained from the hardware detection script, which may also include the processor model, the number of physical cores, etc.

[0038] Understandably, when determining whether a processor is an accelerated processor (such as a GPU, FPGA, ASIC, etc.) or a general-purpose processor (such as a traditional CPU), the first attribute parameter can be compared with the second attribute parameter. If both explicitly indicate that the processor type is accelerated, then the processor can be considered an accelerated processor. For example, if a cloud platform API returns a GPU model, and a hardware detection script also detects the same GPU model inside a virtual machine, then the processor can be considered an accelerated processor. If the first attribute parameter does not mention any identifier for an accelerated processor, or if it does mention one but the second attribute parameter does not, then the processor will be considered a general-purpose processor.

[0039] In the above embodiments, by comparing the first attribute parameter and the second attribute parameter, the processor type of the virtual machine can be accurately determined. This process is crucial for deploying and optimizing data cluster applications in virtual machines. It can ensure the effective allocation and utilization of resources and avoid performance bottlenecks or resource waste caused by improper configuration.

[0040] S206, based on key hardware features, configures the component parameters of the data cluster to be deployed to obtain the configuration file of the data cluster to be deployed;

[0041] It's important to note that data cluster components can include data storage components, such as HDFS (Hadoop Distributed File System), used to store large-scale datasets. Data cluster components can also include resource management and scheduling components, such as YARN (Yet Another Resource Negotiator), responsible for resource allocation and task scheduling. Furthermore, data cluster components can include cluster management and monitoring components, such as Apache ZooKeeper (a distributed coordination service), used to maintain cluster metadata and provide group services, distributed locks, and configuration management; and Apache Spark (a fast and general-purpose engine for large-scale data processing).

[0042] In a data cluster, component parameters refer to the configuration settings used to control and optimize the behavior and performance of these components. These parameters can affect key performance indicators such as data storage strategies, resource allocation methods, computational task execution efficiency, and system fault tolerance. Component parameters are typically set in the component's configuration file and are adjusted according to the cluster's specific hardware configuration, network conditions, and business requirements.

[0043] Specifically, component parameters can include: HDFS parameters such as `dfs.replication` (number of data block replicas) and `dfs.client.read.shortcircuit` (whether to enable short-circuit reads to accelerate read operations on SSD storage nodes). YARN parameters such as `yarn.scheduler.capacity.root.queues` (defining the root queue for resource allocation), `yarn.nodemanager.resource.memory-mb` (available memory size), and `yarn.nodemanager.heartbeat.interval-ms` (heartbeat interval, affecting resource reclamation and fault detection efficiency). Apache Spark parameters such as `spark.executor.memory` (executor memory size) and `spark.cores.max` (maximum number of cores usable in the cluster). Ceph parameters such as `mon osd heartbeat interval` (monitoring heartbeat interval, affecting cluster health status detection) and `osd pool default size` (default number of replicas in the data pool, improving data reliability).

[0044] Understandably, by properly configuring and adjusting these component parameters, the performance, reliability, and cost-effectiveness of data clusters can be significantly improved. When deploying a data cluster, component parameters can be customized based on key hardware characteristics (such as CPU / GPU type, storage type, and network conditions) to obtain configuration files, thereby achieving optimal resource allocation and task scheduling. For example, for nodes with GPUs, YARN and Spark can be configured to identify and optimize GPU resource usage, improving the execution speed of specific types of computing tasks (such as deep learning and image processing).

[0045] Specifically, by adjusting component parameters based on key hardware characteristics, configuration files for the data cluster can be obtained. These configuration files can include hdfs-site.xml, yarn-site.xml, spark-defaults.conf, etc., used to control the behavior and performance of components such as HDFS, YARN, and Spark. On the virtual machine, appropriate file permissions can also be set for the configuration files to ensure that they can only be accessed by the user or process running the data cluster, preventing unauthorized modification or access.

[0046] In some embodiments, after generating the configuration file, its validity can be validated to ensure that the parameters in the configuration file meet the component requirements and do not cause conflicts or anomalies. After validation, automated tools or scripts are used to deploy the configuration file to each node of the data cluster (i.e., to virtual machines) and start the corresponding services. During deployment, the service startup status is monitored to ensure that the cluster components can start and run as expected.

[0047] S208 sends the configuration file to the virtual machine so that the virtual machine can deploy the data cluster to be deployed based on the configuration file.

[0048] A data cluster refers to a group of data nodes that work together to store and process large amounts of data. Each data node performs a specific task within the data cluster, such as storing data, performing data processing tasks, or coordinating the work of other nodes.

[0049] It's important to note that automated deployment scripts (such as Ansible, Chef, and Puppet) running on virtual machines can read and apply configuration files to start and configure various components of the data cluster. Specifically, the deployment script is executed on the virtual machine to begin the deployment process. The script applies parameters from the configuration file and then starts cluster services in a predetermined order, such as the HDFS NameNode, HDFS DataNode, YARN ResourceManager, and YARN NodeManager. During execution, the deployment script continuously monitors the service startup status to ensure all components start successfully according to the specified configuration and order. If any service fails to start, the script should automatically retry or trigger an alert for manual intervention.

[0050] In some embodiments, after the deployment script is completed, or during the deployment process, a health check script or command is executed to verify that the cluster configuration and status meet expectations. For example, verifying whether the YARN GPU nodes are ready, to improve the reliability of the data cluster deployment.

[0051] Through the above embodiments, the configuration file can be accurately and securely transmitted to the virtual machine and applied to the automated deployment process of the data cluster, ensuring that the cluster runs efficiently and stably in the cloud environment, while having good fault tolerance and self-recovery mechanism to adapt to the dynamically changing resource environment.

[0052] Steps S202-S208 above involve obtaining the hardware configuration parameters of the virtual machine; determining the key hardware characteristics of the virtual machine based on the hardware configuration parameters; wherein the key hardware characteristics include at least processor characteristics; furthermore, the component parameters of the components of the data cluster to be deployed can be configured based on the key hardware characteristics to obtain the configuration file of the data cluster to be deployed; finally, the configuration file is sent to the virtual machine so that the virtual machine can deploy the data cluster to be deployed based on the configuration file. In other words, by obtaining the hardware configuration parameters of the virtual machine and determining its processor characteristics, the configuration file of the data cluster can be configured according to the processor characteristics. Since the configuration file takes the processor characteristics into account, the deployment of the data cluster based on the configuration file configured with the processor characteristics can improve the performance of the data cluster.

[0053] In some exemplary embodiments, the component includes a resource management component; step S206 above can be implemented in the following way: when the processor characteristics indicate that the virtual machine's processor is an accelerator, perform at least one of the following configurations on the resource management component to obtain the configuration file of the data cluster to be deployed: determine the resource type file of the resource management component, and set the resource type item in the resource type file to the accelerator; determine the command line tool of the resource management component, and label the command line tool according to the accelerator; determine the task scheduling item of the resource management component, and set the task scheduling item to the accelerator scheduling; determine the resource scheduling item of the resource management component, establish a resource management queue for the resource scheduling item, and set the resource allocation parameters of the resource management queue.

[0054] It's important to note that the resource management component is a software component in a distributed system responsible for managing and allocating hardware resources; this can refer to YARN. The resource type file defines the types of resources, such as GPUs and CPUs. In YARN, to identify and schedule accelerators, resource types can be defined in yarn-site.xml (the resource type file). For example, if a virtual machine has GPU resources (i.e., there are GPU nodes in the data cluster), the resource type item can be set to GPU in the resource type file to ensure that YARN can recognize GPU resources.

[0055] In some embodiments, YARN provides a suite of command-line tools, such as yarn rmadmin, for cluster management and maintenance. To accurately schedule tasks on accelerators, tags can be added to GPU nodes using these command-line tools, allowing YARN to identify nodes with accelerators when scheduling tasks.

[0056] In some embodiments, different queues can be defined for YARN, and corresponding scheduling policies and resource allocation rules can be set. To optimize task execution on accelerator processors, task scheduling can be set to accelerator processor scheduling, that is, ensuring that computationally intensive tasks are preferentially scheduled to GPU nodes.

[0057] In some embodiments, resource scheduling involves how to allocate and schedule resources in a data cluster, particularly in heterogeneous resource environments, and how to efficiently utilize each type of resource. To fully utilize accelerators, resource management queues can be established, and resource allocation parameters can be set, such as the number of GPUs that can be allocated to each task and the priority of GPU tasks. Specifically, resource queues can be defined in yarn-site.xml, and resource allocation parameters can be set.

[0058] In the above embodiments, the resource management component can identify and utilize the characteristics of the accelerator processor to provide more efficient and accurate scheduling strategies for big data tasks. Configuration files can be sent to virtual machines to ensure that the data clusters deployed on the virtual machines can operate according to the resource management strategies in the configuration files.

[0059] In an exemplary embodiment, the key hardware features further include: hard disk features; the components include: storage components; based on the key hardware features, the component parameters of the components of the data cluster to be deployed are configured to obtain the configuration file of the data cluster to be deployed, including: when the hard disk features indicate that the hard disk of the virtual machine is a solid-state drive, determining the short-circuit read parameters of the storage components; setting the short-circuit read parameters to logical true values ​​to obtain the configuration file of the data cluster to be deployed.

[0060] Understandably, in a data cluster, the performance of storage components (such as HDFS) directly impacts the overall efficiency of the cluster. Solid-state drives (SSDs) offer faster read and write speeds and lower latency compared to traditional hard disk drives (HDDs). Therefore, fully utilizing the characteristics of SSDs when configuring storage components can significantly improve data read and write performance.

[0061] It's worth noting that solid-state drives (SSDs) use flash memory chips for data storage, resulting in significantly faster data access speeds compared to hard disk drives (HDDs), especially in random read / write operations. These characteristics of SSDs are highly advantageous for data processing scenarios that require frequent random access and high I / O operations.

[0062] Specifically, HDFS's short-circuit read is an optimization mechanism that allows DataNodes to directly transmit data to clients without going through the NameNode. Enabling this feature on SSDs can significantly reduce network latency and improve data read speeds. For DataNodes running on SSDs, the short-circuit read parameter `dfs.client.read.shortcircuit` can be set to a logical true value to enable short-circuit read functionality. After generating the configuration file, the parameters should be validated to ensure that their format is correct and suitable for SSDs.

[0063] In the above embodiments, dynamic configuration of component parameters based on key hardware characteristics (such as hard drive type) in data cluster deployment can fully utilize hardware advantages and improve data processing performance. For SSD hard drives, optimizing the short-circuit read parameters of storage components can not only reduce network transmission latency but also significantly improve data access speed, making it an effective means to improve cluster performance.

[0064] In an exemplary embodiment, the component includes a resource management component; the method further includes: determining the computing resource mode of the virtual machine; when the computing resource mode is the remaining capacity mode, configuring the component parameters of the resource management component to obtain a configuration file of the data cluster to be deployed; wherein configuring the component parameters of the resource management component includes: determining the heartbeat interval parameter of the resource management component and setting the heartbeat value of the heartbeat interval parameter.

[0065] In a cloud environment, the surplus capacity model refers to cloud service providers offering their unsold, idle computing resources to users at a lower price. This is a cost-effective option for users with fluctuating resource demands who are willing to accept the risk that their resources may be reclaimed at any time. Spot instances are a typical example of the surplus capacity model. Users can use Spot instances at a lower price, but these instances may be reclaimed by the cloud provider when needed.

[0066] The resource management component monitors the status and resource usage of data nodes deployed on virtual machines using heartbeat signals. In remaining capacity mode, since Spot instances may be reclaimed by the cloud service provider at any time, shorter heartbeat intervals can detect abnormal node states more quickly, thereby triggering fault recovery mechanisms faster before resource reclamation.

[0067] Specifically, in YARN's yarn-site.xml, the heartbeat interval parameter is used to set the heartbeat interval time. By default, this interval may be relatively long, but in remaining capacity mode, to improve the system's response speed to node failures, this parameter can be set to a shorter interval, such as 5 seconds or less. A shorter heartbeat interval can detect changes in data node status more quickly, which is suitable for environments using Spot instances.

[0068] In the above embodiments, by setting a heartbeat value, abnormal states, such as alerts for the reclamation of Spot instances, can be detected more promptly in the remaining capacity mode, thereby taking quick measures to avoid data loss and task interruption, and improving the operating efficiency and stability of the data cluster in the cloud environment.

[0069] In an exemplary embodiment, determining the computing resource mode of a virtual machine includes: sending a first query instruction to the virtual machine to obtain mode information returned by the virtual machine; if the mode information is a remaining capacity mode and the duration of sending the first query instruction reaches a preset duration, continuing to send a second query instruction to the virtual machine; and if the mode information returned by the virtual machine based on the second query instruction is a remaining capacity mode, determining that the computing resource mode of the virtual machine is a remaining capacity mode.

[0070] Understandably, the first query command is sent to the virtual machine to inquire about its current compute resource mode. This typically involves querying the type of virtual machine, whether it is an on-demand instance or a remaining capacity mode instance (such as a Spot instance). In cloud platforms, this can be achieved by calling the corresponding API interface. After receiving the first query command, the virtual machine will return its current compute resource mode. If the returned compute resource mode is remaining capacity mode, such as a Spot instance, this indicates that the availability of the virtual machine may be affected by the cloud provider's resource management policies. After receiving the compute resource mode as remaining capacity mode, a preset time can be waited (e.g., a few seconds to a few minutes, the specific time depending on the stability considerations of the cloud platform and the characteristics of Spot instances). This is to ensure that the initial returned mode information is not a temporary state, but rather a stable state of the virtual machine, as the price and availability of Spot instances may change in a short period of time.

[0071] If the virtual machine's compute resource mode remains unchanged within the preset time period, i.e., it is still in the remaining capacity mode, the system will continue to send a second query command to the virtual machine to further confirm the mode information. If the virtual machine confirms the remaining capacity mode again in response to the second query command, then it can be determined that the virtual machine's compute resource mode is indeed in the remaining capacity mode. This step is crucial for deployment strategy because it determines whether additional fault tolerance and optimization mechanisms need to be enabled, such as shortening the heartbeat interval and saving task checkpoints to persistent storage.

[0072] In some embodiments, after determining the virtual machine's compute resource mode, this information can be used to optimize the configuration of resource management components to accommodate the instability of Spot instances. For example, a shorter heartbeat interval can be set in yarn-site.xml, or the number of data replicas can be increased in hdfs-site.xml to improve data fault tolerance. These configuration adjustments are designed to minimize the risk of data loss and task interruption in the event that resources may be suddenly reclaimed.

[0073] In the above embodiments, the computing resource mode of virtual machines can be accurately identified, especially when using the remaining capacity mode (such as Spot instances), ensuring the reliability of the information obtained. This mechanism provides an important foundation for deploying big data clusters in unstable resource environments, allowing administrators and automated systems to optimize configurations based on the type and stability characteristics of virtual machines, thereby improving cluster performance, cost-effectiveness, and overall reliability.

[0074] In an exemplary embodiment, after sending the configuration file to the virtual machine so that the virtual machine can deploy the data cluster to be deployed based on the configuration file, the method further includes: sending a task execution request to the virtual machine so that the corresponding task is executed based on the task execution request on the data cluster deployed on the virtual machine, and obtaining resource reclamation messages at preset time intervals during task execution; wherein, the data cluster is a set of services initialized by the virtual machine based on the configuration file, resulting in multiple services after initialization, and starting the multiple services after initialization in a preset order to complete the deployment of the data cluster to be deployed. If the resource reclamation message indicates that there is remaining capacity and the mode is being reclaimed, the task data is migrated to a copy of the virtual machine.

[0075] After the data cluster deployment is complete, a task execution request is sent to the resource management component (such as YARNResourceManager) on the virtual machine. The task execution request may include specific information about the data processing task to be executed, such as the location of the input data, the processing logic, and the storage location of the output data. Upon receiving the task execution request, the resource management component will schedule the task to an appropriate node for execution based on the current resource usage and configuration. For example, compute-intensive tasks may be scheduled to nodes with GPUs, while data-intensive tasks may be scheduled to the node containing the HDFSDataNode.

[0076] During task execution, resource reclamation information can be queried periodically (at preset intervals, such as every 10 seconds) to understand in real time whether virtual machines (especially Spot instances) have received reclamation notifications from cloud service providers. This is because cloud service providers may decide to reclaim resources in the remaining capacity mode at any time due to market demand, resource management, or other policy factors. Resource reclamation messages can come from the cloud platform's API or notification service. For example, upon receiving a reclamation warning for a Spot instance, applications on the Spot instance can be notified and given a certain amount of time (such as 2 minutes) to clean up and save their state. When a resource reclamation message indicates that resources are being reclaimed, task data migration should be performed. That is, once a resource reclamation message is detected, indicating that the virtual machine (Spot instance) is about to be reclaimed, immediate action is required to protect the running task data and state.

[0077] First, the data of the executing task is saved to persistent storage, such as writing Spark job checkpoints or intermediate results of tasks to HDFS. This step ensures that data is not lost even if the virtual machine is reclaimed. In addition to data migration, the execution state of the task also needs to be saved, including information on the shards being processed and the current processing progress, so that the task can resume from where it left off when restarted. Then, the task can be rescheduled to another available virtual machine node. This node could be another instance in surplus capacity mode or an instance in on-demand mode. During rescheduling, the most suitable node is selected based on the latest resource status and task requirements. On the new node, the task execution is resumed from the saved checkpoints or state information, ensuring the continuity of data processing and minimizing the impact of resource reclamation on task execution.

[0078] In the above embodiments, when a data cluster is running on a virtual machine in the remaining capacity mode on a cloud platform, by monitoring resource reclamation messages in real time and quickly taking data migration strategies upon receiving a reclamation notification, the instability of resources can be effectively addressed, and the continuity of data processing tasks and the integrity of data can be protected.

[0079] The embodiments described above are merely some embodiments of this application, and not all embodiments. To better understand the above methods, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0080] Currently, cloud platforms and big data are developing rapidly, and more and more users are choosing to deploy Hadoop big data clusters on virtual machines on cloud platforms, with deployment methods largely automated. However, when automating the deployment of Hadoop clusters on cloud platforms, existing solutions cannot dynamically perceive underlying hardware differences during the deployment phase. This results in Hadoop failing to fully utilize the potential of heterogeneous computing resources on cloud platforms, with limitations as follows: Traditional deployment tools assume all computing nodes have the same hardware configuration (e.g., pure CPU virtual machines), failing to utilize the heterogeneous resources (GPU instances) provided by the cloud platform. The YARN scheduler for big data components allocates tasks only based on CPU / memory resources, without considering the differences in hardware accelerators (GPUs, etc.), leading to insufficient accelerator utilization. Spot instance compatibility is poor: While using Spot instances can reduce costs, their instability makes it difficult to guarantee the reliability of task execution. Existing solutions do not design task checkpointing mechanisms for the random reclamation characteristics of cloud Spot instances, which can easily cause computational interruptions.

[0081] This application provides a method for deploying a data cluster, referencing Figure 3 The diagram shown illustrates the deployment process of a data cluster, including the following steps:

[0082] S301, Obtain the first configuration parameter, i.e., call the cloud API to obtain the instance specifications. Hardware detection and resource information integration of virtual machines on the cloud platform accurately identify the hardware capabilities of each virtual machine, providing a basis for dynamic configuration. Specifically, by calling the cloud platform's standard API interface, metadata about the virtual machine specifications that can be used to deploy a Hadoop big data cluster can be obtained, and the returned information is recorded. The returned information is the virtual machine's theoretical configuration (i.e., the first configuration parameter), which may include GPU model, number of GPUs, virtual disks, etc. There is an exception handling mechanism during information acquisition. For example, if the API call fails and returns error code 429 (API rate limiting), exponential backoff retries are automatically enabled (initial interval 2 seconds, maximum retries 5 times). If the API still does not return data normally after reaching the maximum number of retries, an error is reported.

[0083] S302, Perform local hardware detection. Execute the following GPU detection script within the target virtual machine to perform hardware detection again, ensuring that the actual hardware configuration of the virtual machine matches the theoretical configuration obtained: Compare the theoretical configuration of the virtual machine with the actual GPU configuration. If the actual configuration matches the theoretical configuration, proceed directly to S303; if the cloud API returns that a GPU exists but is not detected locally, and if the driver is not installed, obtain the virtual machine's operating system version, then pull the corresponding GPU driver based on the operating system version and attempt to install it. After successful installation, perform another detection. If the GPU is detected, proceed to the next step; if the driver installation fails, or the GPU is still not detected after installation, mark the processor type as CPU.

[0084] S303, Determine if the virtual machine's compute resource mode is a Spot instance. Obtain the cloud platform type and verify if the virtual machine's compute resource mode is a Spot instance. Because Spot instances are at risk of being reclaimed at any time, if the virtual machine's compute resource mode is detected as a Spot instance, proceed to S304;

[0085] S304, performs a 300-second warm-up of the virtual machine to ensure instance stability;

[0086] S305, If the virtual machine's computing resource mode is still Spot instance after 300s, then mark the virtual machine is_spot=true(Spot instance);

[0087] S306, if the virtual machine's compute resource mode is not Spot after 300 seconds, then mark the virtual machine is_spot=false (on-demand instance).

[0088] S307, Generate Hardware Profile; Based on key hardware features, generate the final hardware profile. For example, the generated information is as follows:

[0089] node_id":"vm-123" / / Virtual machine ID

[0090] "accelerator" / / object: {

[0091] "type":"GPU", / / Accelerator type

[0092] "model":"Tesla V100", / / model

[0093] "cuda_version":" / / Version number 11.4",

[0094] "count": / / number of accelerators 2

[0095] },

[0096] "storage":{ / / Storage device information for the virtual machine.

[0097] "type": / / Type "SSD",

[0098] "size_gb": / / size500

[0099] },

[0100] "is_spot":true

[0101] }

[0102] S308 generates configuration files; based on the obtained hardware profile, it generates the optimal configuration for deploying key Hadoop components (HDFS, YARN):

[0103] 1) HDFS parameter configuration: If the node has an SSD, enable short-circuit read for HDFS to improve performance. The parameters are as follows:

[0104] <property> / /

[0105] <name> dfs.client.read.shortcircuit< / name> / / This controls whether Short-Circuit Read is enabled when reading HDFS files;

[0106] <value> true< / value> / / Logical truth value;

[0107] < / property> / /

[0108] 2) YARN parameter settings: If any node has a GPU, prioritize modifying yarn-site.xml to enable GPU support, as follows:

[0109] <property>

[0110] <name> yarn.nodemanager.resource-plugins< / name> / / YARN Node Manager Resource Plugin;

[0111] <value> GPU< / value> / / value;

[0112] < / property> / /

[0113] (3) To reduce deployment costs, the big data cluster is prioritized for deployment on Spot instances. In this context, to accommodate the possibility of Spot instances being reclaimed at any time, the YARN heartbeat interval is shortened to 5 seconds to detect Spot instances being reclaimed immediately and to perform fault tolerance processing.

[0114] <property>

[0115] <name> yarn.nodemanager.heartbeat.interval-ms< / name> / / YARN node manager heartbeat interval in milliseconds;

[0116] <value> 5000< / value> / / value;

[0117] < / property> / /

[0118] (4) Tag the GPU nodes:

[0119] `yarn rmadmin-addToClusterNodeLabels"GPU"` / / Adds a label to YARN management.

[0120] yarn rmadmin-replaceLabelsOnNode"vm-123=GPU" / / The node named vm-123 is assigned the label to GPU;

[0121] (5) Add job scheduling rules to Spark to enable its jobs to be automatically scheduled to GPU nodes:

[0122] SparkConf conf = new SparkConf() / / "Creates a new Spark configuration object;

[0123] .set("spark.yarn.resource.tags","accelerator=gpu"); / / Sets the resource tag for Spark during YARN scheduling to "accelerator = GPU";

[0124] (6) Set the scheduler matching rules so that YARN's CapacityScheduler can allocate tasks according to the tags;

[0125] <property> / /

[0126] <name> yarn.scheduler.capacity.root.queues< / name> / / Configuration of the YARN capacity scheduler root queue;

[0127] <value> default, gpu_queue< / value> / / Default queue and GPU queue;

[0128] < / property> / /

[0129] <property> / /

[0130] <name> yarn.scheduler.capacity.root.gpu_queue.accessible-node-labels< / name> / / The node labels accessible to GPU queues under the YARN capacity scheduler root node;

[0131] <value> GPU< / value>

[0132] < / property> / /

[0133] By dynamically configuring the parameters of YARN and SPARK components, YARN and SPARK tasks can be precisely scheduled to the most suitable hardware nodes when the Hadoop big data cluster is working, thereby improving work efficiency.

[0134] S309 verifies the generated configuration file to ensure its validity. Specific verification steps may include: verifying that the configuration information conforms to HDFS, YARN, and Spark parameter format specifications; confirming that the YARN GPU parameters match the actual configuration; if no GPU is present, GPU-related parameters should not exist. If the configuration verification succeeds, proceed to S310; if the configuration verification fails, proceed to S312, generate an alarm message, roll back the configuration, and trigger a manual intervention process. Users can customize the configuration and manually initiate deployment after configuration.

[0135] S310, start the core services. Initiate deployment. After the deployment task starts, check whether the HDFS NameNode is running every 5 seconds. If the HDFS NameNode service is detected to be running, start checking the HDFS DataNode service status and switch them sequentially according to the startup order until all services are detected to be running.

[0136] After successful deployment, execute the corresponding scripts to perform health checks: Verify that the number of live HDFS DataNodes meets expectations: `hdfs dfsadmin-report|grep "Live datanodes"|wc -l`; Verify that the YARN GPU nodes are ready: `yarnnode-list|grep "RUNNING"|grep "GPU"`; Verify the Hadoop cluster functionality through a small job, submitting the test task as follows:

[0137] spark-submit--master yarn / / Submit and run Spark tasks in cluster mode using the YARN scheduler;

[0138] class org.apache.spark.examples.JavaWordCount / / Specifies the class named JavaWordCount to run in the Spark examples;

[0139] / path / to / spark-examples.jar / / Points to the path of the Spark example program JAR file;

[0140] hdfs: / / / input.txt hdfs: / / / output / / The path to the input file in the HDFS file system;

[0141] Result check: Confirm output directory generation: hdfs dfs-ls hdfs: / / / output / _SUCCESS / / Check for success marker;

[0142] Additionally, the system defaults to a maximum waiting time of 20 minutes for each service (this can be configured). If no service is detected as normal after 20 minutes, manual intervention will be required. The startup order of the Hadoop cluster after deployment is set as follows: HDFS NameNode, HDFS DataNode, YARN ResourceManager, YARN NodeManager, Spark History Server, ensuring the data cluster can run normally after deployment and enter S313.

[0143] S311, determine the tasks to be deployed on the Spot instance; refer to Table 1 for the preferred services to deploy on the Spot instance during deployment:

[0144] Table 1

[0145]

[0146] S312, Rollback configuration and alarm. An alarm message is generated, the configuration is rolled back, and the process proceeds to S313;

[0147] S313, triggering manual intervention process;

[0148] S314: Determine if it is a Spot instance. If it is a Spot instance, proceed to S315; otherwise, proceed to S316. Understandably, this is to ensure uninterrupted task execution during Spot instance reclamation, establishing Spot instance fault tolerance and self-healing capabilities to enable task scheduling and automatic migration without user awareness.

[0149] S315, start the Spot monitoring daemon, enter S317;

[0150] S316, standard startup procedure;

[0151] S317 checks for Spot instance reclamation notifications every 10 seconds;

[0152] S318, Has a recycling warning been received? If yes, proceed to S319; if no, proceed to S320.

[0153] S319, immediately triggers hdfs dfs-put${TASK_STATE} / checkpoints / (stores the data of the ongoing task to HDFS);

[0154] S320 is performing the task normally;

[0155] S321, resubmit the task to the new node via the YARN API to complete the migration.

[0156] Finally, update the cluster topology to complete the deployment.

[0157] This application obtains node hardware configurations by abstracting APIs from different cloud platforms and, combined with a two-factor authentication mechanism, has the ability to automatically discover virtual machine hardware accelerators. Based on the virtual machine hardware information, it dynamically sets component parameters within the Hadoop cluster, directing specific computing tasks to the most suitable nodes. Simultaneously, it supports fault-tolerant deployment of Spot instances, fully utilizing the heterogeneous resources of the cloud platform to improve the performance of the deployed cluster. Furthermore, it establishes an automatic awareness and migration scheduling mechanism to address the characteristic of timely recycling of cloud Spot instances, thus improving the method for deploying Hadoop big data clusters on cloud platforms.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0159] Embodiments of this application also provide a parameter adjustment device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the modules described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0160] Figure 4 This is a structural block diagram of a data cluster deployment apparatus according to an embodiment of this application. The apparatus includes:

[0161] The acquisition module 402 is used to acquire the first configuration parameters of the virtual machine; wherein, the first configuration parameters are the hardware configuration parameters of the virtual machine that are set.

[0162] The determining module 404 is configured to determine the key hardware features of the virtual machine based on the first configuration parameters; wherein the key hardware features include at least: processor features;

[0163] Configuration module 406 is used to configure the component parameters of the components of the data cluster to be deployed based on the key hardware features, so as to obtain the configuration file of the data cluster to be deployed.

[0164] Deployment module 408 is used to send the configuration file to the virtual machine, so that the virtual machine can deploy the data cluster to be deployed based on the configuration file.

[0165] The aforementioned device acquires the hardware configuration parameters of the virtual machine; based on the hardware configuration parameters, it determines the key hardware characteristics of the virtual machine; wherein the key hardware characteristics include at least: processor characteristics; further, based on the key hardware characteristics, it configures the component parameters of the data cluster to be deployed, obtaining the configuration file of the data cluster to be deployed; finally, it sends the configuration file to the virtual machine, enabling the virtual machine to deploy the data cluster based on the configuration file. In other words, by acquiring the hardware configuration parameters of the virtual machine and determining its processor characteristics, the configuration file of the data cluster can be configured according to the processor characteristics. Since the configuration file takes the processor characteristics into account, deploying the data cluster based on the configuration file configured with the processor characteristics can improve the performance of the data cluster.

[0166] In an exemplary embodiment, the determining module 404 is further configured to obtain a second configuration parameter of the virtual machine; wherein the second configuration parameter is the actual hardware configuration parameter of the virtual machine detected by the virtual machine's hardware detection script; compare the first configuration parameter with the second configuration parameter to determine the processor type of the virtual machine's processor; and determine the processor characteristics based on the processor type.

[0167] In an exemplary embodiment, the first configuration parameter includes: a first attribute parameter of the processor; the second configuration parameter includes: a second attribute parameter of the processor; the determining module 404 is further configured to compare the first attribute parameter and the second attribute parameter, and if both the first attribute parameter and the second attribute parameter indicate that the processor is an accelerated processor, determine that the processor type is an accelerated processor; otherwise, determine that the processor type is a general-purpose processor.

[0168] In an exemplary embodiment, the component includes a resource management component; the configuration module 406 is further configured to perform at least one of the following configurations on the resource management component to obtain a configuration file for the data cluster to be deployed, when the processor characteristics indicate that the processor of the virtual machine is an accelerator: determining the resource type file of the resource management component and setting the resource type item in the resource type file to the accelerator; determining the command-line tool of the resource management component and tagging the command-line tool according to the accelerator; determining the task scheduling item of the resource management component and setting the task scheduling item to accelerator scheduling; determining the resource scheduling item of the resource management component, establishing a resource management queue for the resource scheduling item, and setting the resource allocation parameters of the resource management queue.

[0169] In an exemplary embodiment, the key hardware features further include: hard disk features; the components include: storage components; the configuration module 406 is further configured to determine the short-circuit read parameters of the storage components when the hard disk features indicate that the hard disk of the virtual machine is a solid-state drive; and set the short-circuit read parameters to logical true values ​​to obtain the configuration file of the data cluster to be deployed.

[0170] In an exemplary embodiment, the component includes a resource management component; the configuration module 406 is further configured to determine the computing resource mode of the virtual machine; when the computing resource mode is the remaining capacity mode, configure the component parameters of the resource management component to obtain the configuration file of the data cluster to be deployed; wherein, configuring the component parameters of the resource management component includes: determining the heartbeat interval parameter of the resource management component and setting the heartbeat value of the heartbeat interval parameter.

[0171] In an exemplary embodiment, the configuration module 406 is further configured to send a first query instruction to the virtual machine to obtain mode information fed back by the virtual machine; if the mode information is a remaining capacity mode and the duration of sending the first query instruction reaches a preset duration, to continue sending a second query instruction to the virtual machine; if the mode information fed back by the virtual machine based on the second query instruction is a remaining capacity mode, to determine that the computing resource mode of the virtual machine is a remaining capacity mode.

[0172] In one exemplary embodiment, the apparatus further includes a migration module; the migration module is configured to send a task execution request to the virtual machine, so that the corresponding task is executed based on the data cluster deployed on the virtual machine according to the task execution request, and during the task execution, obtain resource reclamation messages at preset time intervals; wherein, the data cluster is formed by the virtual machine initializing multiple services of the data cluster to be deployed based on the configuration file, obtaining multiple services after initialization, and starting the multiple services after initialization in a preset order to complete the deployment of the data cluster to be deployed. If the resource reclamation message indicates that there is remaining capacity that needs to be reclaimed, the task data of the task is migrated to a copy of the virtual machine.

[0173] For a description of the features in the embodiment corresponding to the data cluster deployment device, please refer to the relevant description in the embodiment corresponding to the data cluster deployment method, which will not be repeated here.

[0174] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described data cluster deployment method embodiments.

[0175] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described data cluster deployment method embodiments at runtime.

[0176] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0177] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data cluster deployment method embodiments.

[0178] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described data cluster deployment method embodiments.

[0179] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] The deployment method for a data cluster provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for deploying a data cluster, characterized in that, include: Obtain the first configuration parameters of the virtual machine; wherein, the first configuration parameters are the hardware configuration parameters of the virtual machine that are set. Based on the first configuration parameters, the key hardware features of the virtual machine are determined; wherein, the key hardware features include at least: processor features; Based on the aforementioned key hardware features, the component parameters of the components of the data cluster to be deployed are configured to obtain the configuration file of the data cluster to be deployed. The configuration file is sent to the virtual machine so that the virtual machine can deploy the data cluster to be deployed based on the configuration file.

2. The method according to claim 1, characterized in that, The method for determining the processor features includes: Obtain the second configuration parameters of the virtual machine; wherein the second configuration parameters are the actual hardware configuration parameters of the virtual machine detected by the virtual machine's hardware detection script; The first configuration parameter is compared with the second configuration parameter to determine the processor type of the virtual machine's processor; Based on the processor type, the processor characteristics are determined.

3. The method according to claim 2, characterized in that, The first configuration parameter includes: a first attribute parameter of the processor; the second configuration parameter includes: a second attribute parameter of the processor; The step of comparing the first configuration parameter with the second configuration parameter to determine the processor type of the virtual machine includes: The first attribute parameter and the second attribute parameter are compared. If both the first attribute parameter and the second attribute parameter indicate that the processor is an accelerator processor, the processor type is determined to be an accelerator processor. Otherwise, the processor type is determined to be a general-purpose processor.

4. The method according to claim 1, characterized in that, The components include a resource management component; Based on the aforementioned key hardware features, the component parameters of the data cluster to be deployed are configured to obtain the configuration file for the data cluster to be deployed, including: When the processor characteristics indicate that the virtual machine's processor is an accelerated processor, perform at least one of the following configurations on the resource management component to obtain the configuration file for the data cluster to be deployed: Determine the resource type file of the resource management component, and set the resource type item in the resource type file to the acceleration processor; Determine the command-line tools of the resource management component, and label the command-line tools according to the acceleration processor; Determine the task scheduling item of the resource management component and set the task scheduling item to accelerate processor scheduling; Determine the resource scheduling item of the resource management component, establish the resource management queue of the resource scheduling item, and set the resource allocation parameters of the resource management queue.

5. The method according to claim 1, characterized in that, The key hardware features also include: hard disk features; the components include: storage components; Based on the aforementioned key hardware features, the component parameters of the data cluster to be deployed are configured to obtain the configuration file for the data cluster to be deployed, including: If the hard disk characteristics indicate that the virtual machine's hard disk is a solid-state drive, determine the short-circuit read parameters of the storage component; By setting the short-circuit read parameter to a logical true value, the configuration file of the data cluster to be deployed is obtained.

6. The method according to claim 1, characterized in that, The component includes a resource management component; the method further includes: Determine the computing resource mode of the virtual machine; When the computing resource mode is the remaining capacity mode, the component parameters of the resource management component are configured to obtain the configuration file of the data cluster to be deployed; wherein, configuring the component parameters of the resource management component includes: Determine the heartbeat interval parameter of the resource management component, and set the heartbeat value of the heartbeat interval parameter.

7. The method according to claim 6, characterized in that, Determining the computing resource mode of the virtual machine includes: Send a first query command to the virtual machine to obtain the mode information returned by the virtual machine; If the mode information is in the remaining capacity mode and the duration of sending the first query instruction reaches the preset duration, continue to send the second query instruction to the virtual machine; If the virtual machine's mode information based on the second query command is in the remaining capacity mode, then the virtual machine's computing resource mode is determined to be in the remaining capacity mode.

8. The method according to claim 1, characterized in that, After sending the configuration file to the virtual machine so that the virtual machine can deploy the data cluster to be deployed based on the configuration file, the method further includes: A task execution request is sent to the virtual machine, so that the corresponding task is executed based on the data cluster deployed on the virtual machine, and during the task execution, resource reclamation messages are obtained at preset time intervals; wherein, the data cluster is the multiple services of the data cluster to be deployed that the virtual machine initializes based on the configuration file, and the multiple services after initialization are started in a preset order to complete the deployment of the data cluster to be deployed; If the resource reclamation message indicates that there is remaining capacity mode that has been reclaimed, the task data of the task is migrated to a copy of the virtual machine.

9. A data cluster deployment apparatus, characterized in that, include: The acquisition module is used to acquire the first configuration parameters of the virtual machine; wherein, the first configuration parameters are the hardware configuration parameters of the virtual machine that are set. The determining module is configured to determine the key hardware features of the virtual machine based on the first configuration parameters; wherein the key hardware features include at least: processor features; The configuration module is used to configure the component parameters of the components of the data cluster to be deployed based on the key hardware features, so as to obtain the configuration file of the data cluster to be deployed. The deployment module is used to send the configuration file to the virtual machine, so that the virtual machine can deploy the data cluster to be deployed based on the configuration file.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the deployment method of the data cluster as described in any one of claims 1 to 8.

Images